Ceph

On my 0.56.4 cluster, I have some pgs marked as inconsistent because of an omap inconsistency that .4 is able to detect now (see #4743).

I've noticed that "ceph pg scrub" clears the inconsistent flag previously set by deep scrub. Rerunning deep-scrub sets it again, but because of the fact that scrubs run periodically and deep scrubs run less often than simple scrubs, I have seen pgs to flap their inconsistent states.

2013-04-22 10:41:47.809804 osd.95 10.64.0.177:6861/1691 769 : [INF] 3.ba scrub ok
2013-04-22 10:41:52.259532 mon.0 10.64.0.167:6789/0 955160 : [INF] pgmap v5816020: 16760 pgs: 16616 active+clean, 1 active+clean+inconsistent, 143 active+clean+scrubbing+deep; 41514 GB data, 128 TB used, 117 TB / 245 TB avail; 0B/s rd, 0B/s wr, 5op/s
--
2013-04-22 11:55:13.997553 osd.95 10.64.0.177:6861/1691 772 : [ERR] 3.ba osd.134: soid 727dc0ba/.dir.4465.61/head//3 omap_digest 838632401 != known omap_digest 3544866244
2013-04-22 11:55:23.507348 mon.0 10.64.0.167:6789/0 958126 : [INF] pgmap v5818981: 16760 pgs: 16613 active+clean, 1 active+clean+inconsistent, 146 active+clean+scrubbing+deep; 41517 GB data, 128 TB used, 117 TB / 245 TB avail
--
2013-04-22 11:56:07.309062 osd.95 10.64.0.177:6861/1691 773 : [ERR] 3.ba deep-scrub 0 missing, 1 inconsistent objects
2013-04-22 11:56:07.309071 osd.95 10.64.0.177:6861/1691 774 : [ERR] 3.ba deep-scrub 1 errors
2013-04-22 11:56:13.127661 mon.0 10.64.0.167:6789/0 958163 : [INF] pgmap v5819018: 16760 pgs: 16615 active+clean, 2 active+clean+inconsistent, 143 active+clean+scrubbing+deep; 41517 GB data, 128 TB used, 117 TB / 245 TB avail; 15879B/s wr, 3op/s
--
2013-04-22 13:01:56.920412 osd.95 10.64.0.177:6861/1691 776 : [INF] 3.ba scrub ok
2013-04-22 13:02:03.664481 mon.0 10.64.0.167:6789/0 960696 : [INF] pgmap v5821551: 16760 pgs: 16614 active+clean, 1 active+clean+inconsistent, 145 active+clean+scrubbing+deep; 41520 GB data, 128 TB used, 117 TB / 245 TB avail