Ceph » RADOS

symptom:

2018-02-01T12:08:44.630 INFO:tasks.ceph.osd.4.smithi135.stderr:2018-02-01 12:08:44.620649 7f14491a2700 -1 osd.4 pg_epoch: 752 pg[3.2( v 752'2572 (630'2174,752'2572] local-lis/les=751/752 n=50 ec=19/19 lis/c 751/751 les/c/f 752/752/0 726/751/572) [4,2,0] r=0 lpr=751 crt=752'2572 lcod 752'2570 mlcod 752'2570 active+clean+scrubbing] _scan_snaps no head for 3:6702c991:::smithi09729659-280:194 (have MIN)
2018-02-01T12:08:44.632 INFO:tasks.ceph.osd.2.smithi097.stderr:2018-02-01 12:08:44.630462 7fd0eed21700 -1 osd.2 pg_epoch: 752 pg[3.2( v 752'2572 (653'2262,752'2572] local-lis/les=751/752 n=50 ec=19/19 lis/c 751/751 les/c/f 752/752/0 726/751/572) [4,2,0] r=1 lpr=751 luod=0'0 crt=752'2572 lcod 752'2570 active] _scan_snaps no head for 3:6702c991:::smithi09729659-280:194 (have MIN)
2018-02-01T12:08:44.633 INFO:tasks.ceph.osd.0.smithi097.stderr:2018-02-01 12:08:44.631970 7fcccfd05700 -1 osd.0 pg_epoch: 752 pg[3.2( v 752'2572 (630'2174,752'2572] local-lis/les=751/752 n=50 ec=19/19 lis/c 751/751 les/c/f 752/752/0 726/751/572) [4,2,0] r=2 lpr=751 luod=0'0 crt=752'2572 lcod 752'2570 active] _scan_snaps no head for 3:6702c991:::smithi09729659-280:194 (have MIN)

Scrubbing terminated -- not all pgs were active and clean.

scrub will accumulate a cleaned_meta_map that aligns to object boundaries that is used for the PG-type specific scrub step (scrub_snapshot_metadata). this is all good.

however, for each chunk, which does not align to an object boundary (and should not!), we also do _scan_snaps(). This method does the snapmapper repair, and assumes that we have the head object in the same chunk in order to do it. this is not true in general and is basically broken.

the problem unfortunately is that _scan_snaps happens locally on primary and replicas, while the scrub_snapshot_metadata (and accumulation of the cleaned_meta_map) happens only on primary. that means it cannot repair snapmapper on the replica.

we probably need to accumulate cleaned_meta_map on replicas too and similarly gate the _scan_snaps on that...