Bug #11102: mixed version cluster set object digests, then complains on scrub - Ceph - Ceph

Custom queries

Backports: mimic
Backports: missing release
Backports: nautilus
Bluestore
Bug queue
Bug queue - no subprojects
Bug triage
Ceph backlog
Crash queue
Crash triage
Feature Requests
Feedback
My issues
Need Review
Pending backports
Priority queue
Product Backlog Scrub
Project Triage
Test Failures

Actions

Copy link

Bug #11102

closed

mixed version cluster set object digests, then complains on scrub

Added by Yann Dupont about 9 years ago. Updated about 9 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Sage Weil

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

commit:d44f192908dd933354ed2335493f5824469df88a

Hi!

Scenario :
I'm running ceph on armhf (32 bits) boards at home. One of my 3 ARM boards burned. As a temporary leasure, I put orphan disks on a X86_64 machine temporarly. At first, it worked perfectly.
In fact, only for 3 days. Then, scrub started and now I have tons of scrub errors.

If I initiate a scrub on a PG (or even a repair), here are the errors I get :

2015-03-11 20:12:25.383906 osd.2 [ERR] deep-scrub 4.d8 2f9ddfd8/rbd_data.1c262ae8944a.0000000000067dda/head//4 on disk omap digest 0x0 != 0xffffffff
2015-03-11 20:12:25.383933 osd.2 [ERR] deep-scrub 4.d8 1534dfd8/rbd_data.1c262ae8944a.00000000000575ce/head//4 on disk omap digest 0x0 != 0xffffffff

Digest values seem suspicious, and I suppose the root cause is that they are not generated in a arch-neutral way.

BTW : ARM boards are running 0.87 versions, whereas X86_64 board is running 0.93. But I'm not sure it has a special importance in this special case.

History
Notes
Property changes
Associated revisions

Actions

Copy link

Updated by Yann Dupont about 9 years ago

Read "As a temporary measure" , not "temporary leasure" of course (event if this had some sense... :D )

BTW : here is the crushmap

dupont-y@duponty-desktop:~$ ceph osd tree

id weight type name up/down reweight
-12 5.784 root USB
-9 2.926 host OSD-B1-USB
0 1.46 osd.0 up 1
2 0.49 osd.2 up 1
8 0.976 osd.8 up 1
-10 1.41 host OSD-B2-USB
1 1.41 osd.1 up 1
-11 1.448 host OSD-B3-USB
3 0.976 osd.3 up 1
7 0.471 osd.7 up 1
-5 0.15 root SSD
-6 0.04999 host OSD-B1-SSD
4 0.04999 osd.4 down 0
-7 0.04999 host OSD-B2-SSD
5 0.04999 osd.5 up 1
-8 0.04999 host OSD-B3-SSD
6 0.04999 osd.6 up 1
-1 0 root default

OSD-B1-USB is now X86_64 platform (and not USB but SATA BTW)

OSD-B2-USB & OSD-B3-USB are armhf platforms.

And AFAICS scrub errors occurs when primary and secondary are on different ARCH.

HEALTH_ERR 110 pgs inconsistent; 3 near full osd(s); 41192 scrub errors; noscrub,nodeep-scrub flag(s) set; mon.MON-B1 low disk space
pg 4.d3 is active+clean+inconsistent, acting [8,1]
pg 4.d0 is active+clean+inconsistent, acting [2,1]
pg 4.d6 is active+clean+inconsistent, acting [0,7]
pg 4.d5 is active+clean+inconsistent, acting [8,1]
pg 4.c9 is active+clean+inconsistent, acting [0,3]
pg 4.c8 is active+clean+inconsistent, acting [0,1]
pg 4.cd is active+clean+inconsistent, acting [8,1]
pg 4.cc is active+clean+inconsistent, acting [8,1]
pg 4.c3 is active+clean+inconsistent, acting [0,1]
pg 4.c1 is active+clean+inconsistent, acting [0,1]
pg 4.c0 is active+clean+inconsistent, acting [8,1]
pg 4.c7 is active+clean+inconsistent, acting [0,1]
pg 4.c5 is active+clean+inconsistent, acting [8,1]
pg 4.c4 is active+clean+inconsistent, acting [2,3]
pg 4.bb is active+clean+inconsistent, acting [0,7]
pg 4.bf is active+clean+inconsistent, acting [0,3]
pg 4.be is active+clean+inconsistent, acting [0,3]
pg 4.bc is active+clean+inconsistent, acting [0,3]
pg 4.b3 is active+clean+inconsistent, acting [8,1]
pg 4.b2 is active+clean+inconsistent, acting [2,1]
pg 4.b1 is active+clean+inconsistent, acting [0,7]
pg 4.b6 is active+clean+inconsistent, acting [0,3]
pg 4.b5 is active+clean+inconsistent, acting [0,1]
pg 4.ab is active+clean+inconsistent, acting [8,1]
pg 4.aa is active+clean+inconsistent, acting [0,3]
pg 4.a8 is active+clean+inconsistent, acting [0,3]
pg 4.a2 is active+clean+inconsistent, acting [8,1]
pg 4.a7 is active+clean+inconsistent, acting [0,3]
pg 4.a5 is active+clean+inconsistent, acting [0,3]
pg 4.99 is active+clean+inconsistent, acting [8,1]
pg 4.9f is active+clean+inconsistent, acting [0,3]
pg 4.93 is active+clean+inconsistent, acting [2,1]
pg 4.90 is active+clean+inconsistent, acting [0,1]
pg 4.8a is active+clean+inconsistent, acting [0,3]
pg 4.8d is active+clean+inconsistent, acting [8,3]
pg 4.86 is active+clean+inconsistent, acting [8,1]
pg 4.85 is active+clean+inconsistent, acting [2,1]
pg 4.78 is active+clean+inconsistent, acting [0,3]
pg 4.7f is active+clean+inconsistent, acting [8,7]
pg 4.73 is active+clean+inconsistent, acting [8,1]
pg 4.71 is active+clean+inconsistent, acting [0,3]
pg 4.77 is active+clean+inconsistent, acting [0,3]
pg 4.75 is active+clean+inconsistent, acting [2,1]
pg 4.6b is active+clean+inconsistent, acting [8,3]
pg 4.6a is active+clean+inconsistent, acting [8,1]
pg 4.6e is active+clean+inconsistent, acting [2,1]
pg 4.6d is active+clean+inconsistent, acting [0,3]
pg 4.60 is active+clean+inconsistent, acting [2,3]
pg 4.67 is active+clean+inconsistent, acting [0,1]
pg 4.66 is active+clean+inconsistent, acting [8,7]
pg 4.65 is active+clean+inconsistent, acting [0,3]
pg 4.59 is active+clean+inconsistent, acting [0,3]
pg 4.58 is active+clean+inconsistent, acting [0,3]
pg 4.5f is active+clean+inconsistent, acting [2,3]
pg 4.5e is active+clean+inconsistent, acting [8,3]
pg 4.5d is active+clean+inconsistent, acting [0,3]
pg 4.53 is active+clean+inconsistent, acting [8,3]
pg 4.51 is active+clean+inconsistent, acting [0,3]
pg 4.57 is active+clean+inconsistent, acting [8,7]
pg 4.56 is active+clean+inconsistent, acting [0,1]
pg 4.4e is active+clean+inconsistent, acting [2,1]
pg 4.4c is active+clean+inconsistent, acting [8,3]
pg 4.43 is active+clean+inconsistent, acting [0,1]
pg 4.41 is active+clean+inconsistent, acting [8,1]
pg 4.40 is active+clean+inconsistent, acting [8,1]
pg 4.46 is active+clean+inconsistent, acting [0,3]
pg 4.3a is active+clean+inconsistent, acting [8,1]
pg 4.38 is active+clean+inconsistent, acting [0,1]
pg 4.3d is active+clean+inconsistent, acting [0,1]
pg 4.37 is active+clean+inconsistent, acting [0,3]
pg 4.34 is active+clean+inconsistent, acting [0,3]
pg 4.2b is active+clean+inconsistent, acting [0,1]
pg 4.2a is active+clean+inconsistent, acting [0,7]
pg 4.26 is active+clean+inconsistent, acting [8,3]
pg 4.25 is active+clean+inconsistent, acting [8,1]
pg 4.24 is active+clean+inconsistent, acting [0,1]
pg 4.1a is active+clean+inconsistent, acting [8,3]
pg 4.18 is active+clean+inconsistent, acting [8,3]
pg 4.1e is active+clean+inconsistent, acting [0,7]
pg 4.1c is active+clean+inconsistent, acting [2,1]
pg 4.17 is active+clean+inconsistent, acting [0,3]
pg 4.15 is active+clean+inconsistent, acting [0,1]
pg 4.a is active+clean+inconsistent, acting [8,1]
pg 4.9 is active+clean+inconsistent, acting [0,1]
pg 4.8 is active+clean+inconsistent, acting [8,3]
pg 4.e is active+clean+inconsistent, acting [8,7]
pg 4.c is active+clean+inconsistent, acting [8,1]
pg 4.3 is active+clean+inconsistent, acting [0,3]
pg 4.5 is active+clean+inconsistent, acting [8,1]
pg 4.4 is active+clean+inconsistent, acting [0,1]
pg 4.fb is active+clean+inconsistent, acting [8,3]
pg 4.fa is active+clean+inconsistent, acting [8,3]
pg 4.f8 is active+clean+inconsistent, acting [0,1]
pg 4.ff is active+clean+inconsistent, acting [2,3]
pg 4.fe is active+clean+inconsistent, acting [0,1]
pg 4.fc is active+clean+inconsistent, acting [0,7]
pg 4.f7 is active+clean+inconsistent, acting [8,3]
pg 4.f6 is active+clean+inconsistent, acting [0,3]
pg 4.f5 is active+clean+inconsistent, acting [0,7]
pg 4.ea is active+clean+inconsistent, acting [0,3]
pg 4.ee is active+clean+inconsistent, acting [0,7]
pg 4.ed is active+clean+inconsistent, acting [0,1]
pg 4.ec is active+clean+inconsistent, acting [0,1]
pg 4.e2 is active+clean+inconsistent, acting [2,1]
pg 4.e0 is active+clean+inconsistent, acting [8,7]
pg 4.e5 is active+clean+inconsistent, acting [0,3]
pg 4.db is active+clean+inconsistent, acting [8,3]
pg 4.da is active+clean+inconsistent, acting [8,3]
pg 4.d9 is active+clean+inconsistent, acting [8,1]
pg 4.d8 is active+clean+inconsistent, acting [2,1]

Actions

Copy link

Updated by Loïc Dachary about 9 years ago

Priority changed from Normal to High

I heard of a mixed armhf / x86 cluster suffering from problems as well, last week. I don't have details though.

Actions

Copy link

Updated by Sage Weil about 9 years ago

Subject changed from Omap digest probably not generated in arch neutral way. to mixed version cluster set object digests, then complains on scrub
Assignee set to Sage Weil
Priority changed from High to Urgent
Source changed from other to Q/A

this is a mixed version problem:

1. primary is new osd that sets a digest because it is a trivial op (create, write_full, append)
2. replicas do not record it (they are old)
3. scrub complains because primary has digest but replicas do not report it properly

this is confounded by the fact that we changed the initial value from 0 to -1 at the same time.

Actions

Copy link