Project

General

Profile

Bug #11102

mixed version cluster set object digests, then complains on scrub

Added by Yann Dupont about 9 years ago. Updated about 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

commit:d44f192908dd933354ed2335493f5824469df88a

Hi!

Scenario :
I'm running ceph on armhf (32 bits) boards at home. One of my 3 ARM boards burned. As a temporary leasure, I put orphan disks on a X86_64 machine temporarly. At first, it worked perfectly.
In fact, only for 3 days. Then, scrub started and now I have tons of scrub errors.

If I initiate a scrub on a PG (or even a repair), here are the errors I get :

2015-03-11 20:12:25.383906 osd.2 [ERR] deep-scrub 4.d8 2f9ddfd8/rbd_data.1c262ae8944a.0000000000067dda/head//4 on disk omap digest 0x0 != 0xffffffff
2015-03-11 20:12:25.383933 osd.2 [ERR] deep-scrub 4.d8 1534dfd8/rbd_data.1c262ae8944a.00000000000575ce/head//4 on disk omap digest 0x0 != 0xffffffff

Digest values seem suspicious, and I suppose the root cause is that they are not generated in a arch-neutral way.

BTW : ARM boards are running 0.87 versions, whereas X86_64 board is running 0.93. But I'm not sure it has a special importance in this special case.

Associated revisions

Revision 477ac929 (diff)
Added by Sage Weil about 9 years ago

osd: only complain about stored vs actual digest if all peers support it

If we have a mixed cluster of hammer and pre-hammer OSDs, we will fall back
to using 0 as the initial crc32c value. However, if the primary has a
stored digest, it currently compares its value to the reported value (w/
the wrong initial value) and complains.

There are two possible fixes:
- avoid storing a digest if all peers don't support it, or
- avoid complaining on scrub if all peers don't support it.

The latter is easier, and this fix also has the benefit of fixing the bug
even for clusters where this has already happened.

Fixes: #11102
Signed-off-by: Sage Weil <>

History

#1 Updated by Yann Dupont about 9 years ago

Read "As a temporary measure" , not "temporary leasure" of course (event if this had some sense... :D )

BTW : here is the crushmap

dupont-y@duponty-desktop:~$ ceph osd tree
  1. id weight type name up/down reweight
    -12 5.784 root USB
    -9 2.926 host OSD-B1-USB
    0 1.46 osd.0 up 1
    2 0.49 osd.2 up 1
    8 0.976 osd.8 up 1
    -10 1.41 host OSD-B2-USB
    1 1.41 osd.1 up 1
    -11 1.448 host OSD-B3-USB
    3 0.976 osd.3 up 1
    7 0.471 osd.7 up 1
    -5 0.15 root SSD
    -6 0.04999 host OSD-B1-SSD
    4 0.04999 osd.4 down 0
    -7 0.04999 host OSD-B2-SSD
    5 0.04999 osd.5 up 1
    -8 0.04999 host OSD-B3-SSD
    6 0.04999 osd.6 up 1
    -1 0 root default

OSD-B1-USB is now X86_64 platform (and not USB but SATA BTW)

OSD-B2-USB & OSD-B3-USB are armhf platforms.

And AFAICS scrub errors occurs when primary and secondary are on different ARCH.

HEALTH_ERR 110 pgs inconsistent; 3 near full osd(s); 41192 scrub errors; noscrub,nodeep-scrub flag(s) set; mon.MON-B1 low disk space
pg 4.d3 is active+clean+inconsistent, acting [8,1]
pg 4.d0 is active+clean+inconsistent, acting [2,1]
pg 4.d6 is active+clean+inconsistent, acting [0,7]
pg 4.d5 is active+clean+inconsistent, acting [8,1]
pg 4.c9 is active+clean+inconsistent, acting [0,3]
pg 4.c8 is active+clean+inconsistent, acting [0,1]
pg 4.cd is active+clean+inconsistent, acting [8,1]
pg 4.cc is active+clean+inconsistent, acting [8,1]
pg 4.c3 is active+clean+inconsistent, acting [0,1]
pg 4.c1 is active+clean+inconsistent, acting [0,1]
pg 4.c0 is active+clean+inconsistent, acting [8,1]
pg 4.c7 is active+clean+inconsistent, acting [0,1]
pg 4.c5 is active+clean+inconsistent, acting [8,1]
pg 4.c4 is active+clean+inconsistent, acting [2,3]
pg 4.bb is active+clean+inconsistent, acting [0,7]
pg 4.bf is active+clean+inconsistent, acting [0,3]
pg 4.be is active+clean+inconsistent, acting [0,3]
pg 4.bc is active+clean+inconsistent, acting [0,3]
pg 4.b3 is active+clean+inconsistent, acting [8,1]
pg 4.b2 is active+clean+inconsistent, acting [2,1]
pg 4.b1 is active+clean+inconsistent, acting [0,7]
pg 4.b6 is active+clean+inconsistent, acting [0,3]
pg 4.b5 is active+clean+inconsistent, acting [0,1]
pg 4.ab is active+clean+inconsistent, acting [8,1]
pg 4.aa is active+clean+inconsistent, acting [0,3]
pg 4.a8 is active+clean+inconsistent, acting [0,3]
pg 4.a2 is active+clean+inconsistent, acting [8,1]
pg 4.a7 is active+clean+inconsistent, acting [0,3]
pg 4.a5 is active+clean+inconsistent, acting [0,3]
pg 4.99 is active+clean+inconsistent, acting [8,1]
pg 4.9f is active+clean+inconsistent, acting [0,3]
pg 4.93 is active+clean+inconsistent, acting [2,1]
pg 4.90 is active+clean+inconsistent, acting [0,1]
pg 4.8a is active+clean+inconsistent, acting [0,3]
pg 4.8d is active+clean+inconsistent, acting [8,3]
pg 4.86 is active+clean+inconsistent, acting [8,1]
pg 4.85 is active+clean+inconsistent, acting [2,1]
pg 4.78 is active+clean+inconsistent, acting [0,3]
pg 4.7f is active+clean+inconsistent, acting [8,7]
pg 4.73 is active+clean+inconsistent, acting [8,1]
pg 4.71 is active+clean+inconsistent, acting [0,3]
pg 4.77 is active+clean+inconsistent, acting [0,3]
pg 4.75 is active+clean+inconsistent, acting [2,1]
pg 4.6b is active+clean+inconsistent, acting [8,3]
pg 4.6a is active+clean+inconsistent, acting [8,1]
pg 4.6e is active+clean+inconsistent, acting [2,1]
pg 4.6d is active+clean+inconsistent, acting [0,3]
pg 4.60 is active+clean+inconsistent, acting [2,3]
pg 4.67 is active+clean+inconsistent, acting [0,1]
pg 4.66 is active+clean+inconsistent, acting [8,7]
pg 4.65 is active+clean+inconsistent, acting [0,3]
pg 4.59 is active+clean+inconsistent, acting [0,3]
pg 4.58 is active+clean+inconsistent, acting [0,3]
pg 4.5f is active+clean+inconsistent, acting [2,3]
pg 4.5e is active+clean+inconsistent, acting [8,3]
pg 4.5d is active+clean+inconsistent, acting [0,3]
pg 4.53 is active+clean+inconsistent, acting [8,3]
pg 4.51 is active+clean+inconsistent, acting [0,3]
pg 4.57 is active+clean+inconsistent, acting [8,7]
pg 4.56 is active+clean+inconsistent, acting [0,1]
pg 4.4e is active+clean+inconsistent, acting [2,1]
pg 4.4c is active+clean+inconsistent, acting [8,3]
pg 4.43 is active+clean+inconsistent, acting [0,1]
pg 4.41 is active+clean+inconsistent, acting [8,1]
pg 4.40 is active+clean+inconsistent, acting [8,1]
pg 4.46 is active+clean+inconsistent, acting [0,3]
pg 4.3a is active+clean+inconsistent, acting [8,1]
pg 4.38 is active+clean+inconsistent, acting [0,1]
pg 4.3d is active+clean+inconsistent, acting [0,1]
pg 4.37 is active+clean+inconsistent, acting [0,3]
pg 4.34 is active+clean+inconsistent, acting [0,3]
pg 4.2b is active+clean+inconsistent, acting [0,1]
pg 4.2a is active+clean+inconsistent, acting [0,7]
pg 4.26 is active+clean+inconsistent, acting [8,3]
pg 4.25 is active+clean+inconsistent, acting [8,1]
pg 4.24 is active+clean+inconsistent, acting [0,1]
pg 4.1a is active+clean+inconsistent, acting [8,3]
pg 4.18 is active+clean+inconsistent, acting [8,3]
pg 4.1e is active+clean+inconsistent, acting [0,7]
pg 4.1c is active+clean+inconsistent, acting [2,1]
pg 4.17 is active+clean+inconsistent, acting [0,3]
pg 4.15 is active+clean+inconsistent, acting [0,1]
pg 4.a is active+clean+inconsistent, acting [8,1]
pg 4.9 is active+clean+inconsistent, acting [0,1]
pg 4.8 is active+clean+inconsistent, acting [8,3]
pg 4.e is active+clean+inconsistent, acting [8,7]
pg 4.c is active+clean+inconsistent, acting [8,1]
pg 4.3 is active+clean+inconsistent, acting [0,3]
pg 4.5 is active+clean+inconsistent, acting [8,1]
pg 4.4 is active+clean+inconsistent, acting [0,1]
pg 4.fb is active+clean+inconsistent, acting [8,3]
pg 4.fa is active+clean+inconsistent, acting [8,3]
pg 4.f8 is active+clean+inconsistent, acting [0,1]
pg 4.ff is active+clean+inconsistent, acting [2,3]
pg 4.fe is active+clean+inconsistent, acting [0,1]
pg 4.fc is active+clean+inconsistent, acting [0,7]
pg 4.f7 is active+clean+inconsistent, acting [8,3]
pg 4.f6 is active+clean+inconsistent, acting [0,3]
pg 4.f5 is active+clean+inconsistent, acting [0,7]
pg 4.ea is active+clean+inconsistent, acting [0,3]
pg 4.ee is active+clean+inconsistent, acting [0,7]
pg 4.ed is active+clean+inconsistent, acting [0,1]
pg 4.ec is active+clean+inconsistent, acting [0,1]
pg 4.e2 is active+clean+inconsistent, acting [2,1]
pg 4.e0 is active+clean+inconsistent, acting [8,7]
pg 4.e5 is active+clean+inconsistent, acting [0,3]
pg 4.db is active+clean+inconsistent, acting [8,3]
pg 4.da is active+clean+inconsistent, acting [8,3]
pg 4.d9 is active+clean+inconsistent, acting [8,1]
pg 4.d8 is active+clean+inconsistent, acting [2,1]

#2 Updated by Loïc Dachary about 9 years ago

  • Priority changed from Normal to High

I heard of a mixed armhf / x86 cluster suffering from problems as well, last week. I don't have details though.

#3 Updated by Sage Weil about 9 years ago

  • Subject changed from Omap digest probably not generated in arch neutral way. to mixed version cluster set object digests, then complains on scrub
  • Assignee set to Sage Weil
  • Priority changed from High to Urgent
  • Source changed from other to Q/A

this is a mixed version problem:

1. primary is new osd that sets a digest because it is a trivial op (create, write_full, append)
2. replicas do not record it (they are old)
3. scrub complains because primary has digest but replicas do not report it properly

this is confounded by the fact that we changed the initial value from 0 to -1 at the same time.

#4 Updated by Sage Weil about 9 years ago

  • Status changed from New to Resolved

#5 Updated by Yann Dupont about 9 years ago

OK, I was totally wrong in the root cause . Glad to see this patch, thanks !

#6 Updated by Loïc Dachary about 9 years ago

  • Description updated (diff)

Also available in: Atom PDF