Project

General

Profile

Bug #4743

omap deep scrub finds multiple PGs as inconsistent

Added by Faidon Liambotis almost 11 years ago. Updated almost 11 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I've inquired on IRC a few times about this (my nickname is paravoid) but filing it here since it sounds serious enough. I'm setting urgency to "high" as this might be a data corruption bug.

I have a cluster with 144 (135 in) OSDs in 12 boxes, all running Ubuntu 12.04 LTS and using XFS exclusively. The cluster started as Ceph 0.54 and has progressed over time to 0.55, 0.56 and all the 0.56.x releases. ceph.conf is more or less stock, with almost no settings at all.

Since the 0.56.4 upgrade I've been getting a lot of inconsistent PGs, all due to omap inconsistencies. Since the omap deep scrubbing in new in .4, the working theory from #ceph was that this is due to a previous version's bug that corrupted those omaps and that has since been fixed.

So what I did, was run a deep scrub across all PGs (~16k); this found 64 PGs as inconsistent, all over the cluster's OSDs. I subsequently ran a repair on them one by one over the weekend as suggested by Sam and the cluster happily returned to HEALTH_OK.

Since Monday though, more PGs have cropped up as inconsistent, so far PGs that were previously marked as okay (9 so far). Repeating the deep scrub for an inconsistent pg results in the exact same error, so if it's a deep scrub bug, it's reproducible.

Example, for:
3.2f2 15195 0 0 0 2695504959 135977 135977 active+clean+inconsistent 2013-04-16 18:10:18.046568 171935'1519876 171775'1791475 [56,86,133] [56,86,133] 171935'1516874 2013-04-16 18:10:18.046524 171935'1516874 2013-04-16 18:10:18.046524

2013-04-11 11:51:22.376049 osd.56 10.64.32.10:6857/17945 232 : [INF] 3.2f2 scrub ok
2013-04-14 19:19:10.827868 osd.56 10.64.32.10:6857/17945 342 : [INF] 3.2f2 scrub ok
2013-04-16 16:42:40.219647 osd.56 10.64.32.10:6857/17945 409 : [ERR] 3.2f2 osd.133: soid d340c2f2/.dir.10267.612/head//3 omap_digest 4263226353 != known omap_digest 3162369895
2013-04-16 18:04:25.041373 osd.56 10.64.32.10:6857/17945 414 : [ERR] 3.2f2 osd.133: soid d340c2f2/.dir.10267.612/head//3 omap_digest 4263226353 != known omap_digest 3162369895

Sam suggested to rerun a deep scrub on one of the pgs with debug filestore = 20 debug osd = 20 debug ms = 1 on primary/replicas for one pg. I'll proceed to do that as soon as the cluster-wide deep scrub finishes (~a day or two) or it'll be too noisy otherwise.

History

#1 Updated by Faidon Liambotis almost 11 years ago

I got debug filestore = 20 debug osd = 30 debug ms = 1 (turns out it needs 30, not 20) logs from all three replicas of such a PG while rerunning a deep scrub on it and gave them to Sam.

#2 Updated by Samuel Just almost 11 years ago

osd.133 is missing 3 keys (out of 750k) on object 3.2f2_head/d340c2f2/.dir.10267.612/head//3

#3 Updated by Samuel Just almost 11 years ago

the xattrs, however, seem to match

#4 Updated by Samuel Just almost 11 years ago

  • Status changed from New to Can't reproduce
  • Assignee set to Samuel Just

I think this was actually caused by one of the journal replay defects from <56.4. I'm marking it can't reproduce until we get evidence of a new occurrence.

Also available in: Atom PDF