Project

General

Profile

Bug #13862

pgs stuck inconsistent after infernalis upgrade

Added by Logan V over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
11/23/2015
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
infernalis
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

During my infernalis upgrade I was running unmatched OSD versions for about 48 hours while I did a rolling chown/upgrade across ~30 hosts and ~200 OSDs. I noticed during the upgrade process that some pgs were going inconsistent. After finishing the upgrade completely across all OSDs I had about 80 pgs marked inconsistent, all in the same EC pool.

I did a mass repair across these 80 pgs and many of them completed repair successfully.

However, 36 did not repair and failed with the same error message. These 36 pgs have been stuck inconsistent for about a week now.

The inconsistent pgs:

# ceph health detail
HEALTH_ERR 36 pgs inconsistent; 80 scrub errors; noout flag(s) set
pg 33.f62 is active+clean+inconsistent, acting [143,136,77,39]
pg 33.e6c is active+clean+inconsistent, acting [133,105,74,67]
pg 33.e02 is active+clean+inconsistent, acting [67,114,104,91]
pg 33.d97 is active+clean+inconsistent, acting [108,68,58,101]
pg 33.d62 is active+clean+inconsistent, acting [138,100,46,141]
pg 33.d47 is active+clean+inconsistent, acting [89,23,69,77]
pg 33.caa is active+clean+inconsistent, acting [99,84,79,95]
pg 33.ca1 is active+clean+inconsistent, acting [88,144,108,107]
pg 33.c73 is active+clean+inconsistent, acting [94,66,90,116]
pg 33.c06 is active+clean+inconsistent, acting [135,116,63,138]
pg 33.be0 is active+clean+inconsistent, acting [137,112,102,82]
pg 33.bae is active+clean+inconsistent, acting [133,104,94,101]
pg 33.b48 is active+clean+inconsistent, acting [135,95,77,72]
pg 33.b21 is active+clean+inconsistent, acting [98,67,83,62]
pg 33.a5f is active+clean+inconsistent, acting [134,131,95,96]
pg 33.a46 is active+clean+inconsistent, acting [133,138,105,55]
pg 33.7b5 is active+clean+inconsistent, acting [110,63,75,83]
pg 33.3bf is active+clean+inconsistent, acting [135,98,116,88]
pg 33.329 is active+clean+inconsistent, acting [103,91,108,132]
pg 33.1ac is active+clean+inconsistent, acting [138,34,105,128]
pg 33.12b is active+clean+inconsistent, acting [81,79,100,74]
pg 33.279 is active+clean+inconsistent, acting [95,105,34,68]
pg 33.497 is active+clean+inconsistent, acting [90,63,129,94]
pg 33.537 is active+clean+inconsistent, acting [141,108,24,72]
pg 33.518 is active+clean+inconsistent, acting [95,140,67,79]
pg 33.5a9 is active+clean+inconsistent, acting [116,79,143,138]
pg 33.701 is active+clean+inconsistent, acting [96,40,140,104]
pg 33.594 is active+clean+inconsistent, acting [72,93,81,111]
pg 33.5d1 is active+clean+inconsistent, acting [81,133,77,136]
pg 33.632 is active+clean+inconsistent, acting [114,112,86,18]
pg 33.610 is active+clean+inconsistent, acting [112,142,103,68]
pg 33.6b5 is active+clean+inconsistent, acting [143,135,139,84]
pg 33.768 is active+clean+inconsistent, acting [89,92,108,24]
pg 33.8ab is active+clean+inconsistent, acting [95,138,99,116]
pg 33.8c4 is active+clean+inconsistent, acting [50,111,98,100]
pg 33.913 is active+clean+inconsistent, acting [74,138,113,72]

Output from ceph -w when I tell one of these pg's to repair:

2015-11-23 10:26:28.864457 mon.1 [INF] from='client.? 10.10.7.10:0/2688742338' entity='client.admin' cmd=[{"prefix": "pg repair", "pgid": "33.f62"}]: dispatch
2015-11-23 10:26:31.383884 osd.143 [INF] 33.f62 repair starts
2015-11-23 10:26:33.209568 osd.143 [ERR] 33.f62s0 shard 4(0): soid failed to pick suitable auth object
2015-11-23 10:26:33.209861 osd.143 [ERR] repair 33.f62s0 -1/00000000/temp_33.f62s0_0_93345312_70/head no 'snapset' attr

I have tried the following process to resolve this as well:
1) Stopped the OSD failing the repair with this error
2) Delete the inconsistent pg from the osd
3) Restart the OSD
4) Repair the pg again, usually the repair will succeed now
Then after waiting several hours the pg will be remarked as inconsistent.

This is a dev pool on older drives, older hosts, etc. so I cannot rule out some host issue causing this, but I notice that the inconsistent pgs do not appear to have any particular osds in common as I would usually see with host issues in the past. I would be happy to perform additional debugging, but I need a little guidance on what would be useful.


Related issues

Related to Ceph - Bug #13381: osd/SnapMapper.cc: 282: FAILED assert(check(oid)) on hammer->jewel upgrade New 10/06/2015
Copied to Ceph - Backport #14494: infernalis: pgs stuck inconsistent after infernalis upgrade Resolved

Associated revisions

Revision 10b4a082 (diff)
Added by David Zafman over 3 years ago

osd/OSD: clear_temp_objects() include removal of Hammer temp objects

Fixes: #13862

Signed-off-by: David Zafman <>

Revision 2480fca9 (diff)
Added by David Zafman over 3 years ago

osd/OSD: clear_temp_objects() include removal of Hammer temp objects

Fixes: #13862

Signed-off-by: David Zafman <>
(cherry picked from commit 10b4a0825d9917b6fdd0d6450640238b78ba05d4)

History

#1 Updated by Sage Weil over 3 years ago

  • Priority changed from Normal to Urgent

#2 Updated by David Zafman over 3 years ago

This is a hammer temp object which no longer applies in Infernalis.

-1/00000000/temp_33.f62s0_0_93345312_70/head

In Infernalis the internal temp objects use a pool value of <= -2. So after the upgrade I must presume that we didn't clean these objects up. This is causing the inconsistent PGs. It doesn't look like anything harmful, but we should put in a fix to get these cleaned up so that the PGs can get out of inconsistent state.

#3 Updated by Logan V over 3 years ago

Interesting.. thanks for the info. In advance of a fix, should I be able to fix these pgs by removing the obsolete temp objects manually on the osd filesystems housing these pgs?

#4 Updated by David Zafman over 3 years ago

Logan can you find the temp_33.f62s0_0_93345312_70 in the filestore and tell me the path inside the filestore and actual file name? The first part of the name probably looks like this: temp\u33.f62s0\u0\u93345312\u70__head_00000000.

#5 Updated by David Zafman over 3 years ago

  • Status changed from New to In Progress
  • Backport set to Infernalis

I'm creating a pull request for Jewel release which will clean out these stray Hammer temp files on OSD startup. This will need to be backported to Infernalis.

#6 Updated by David Zafman over 3 years ago

  • Status changed from In Progress to Need Review

#7 Updated by David Zafman over 3 years ago

  • Assignee set to David Zafman

#8 Updated by Logan V over 3 years ago

Looks like 33.f62 is no longer showing up as inconsistent. Maybe some operation performed on the pg over the last few weeks has fixed it. There are still 26 inconsistent. Here is one:

pg 33.e02 is active+clean+inconsistent, acting [67,114,104,91]

$ for i in osd-67 osd-114 osd-104 osd-91; do echo "checking on $i"; s root@$i 'ls -l /var/lib/ceph/osd/ceph-*/current/33.e02*_TEMP'; done
checking on osd-67
total 2228
-rw-r--r-- 1 ceph ceph  879264 Nov 10 20:03 temp\u33.e02s0\u0\u93385203\u2__head_00000000__none_ffffffffffffffff_0
-rw-r--r-- 1 ceph ceph 1399392 Nov 20 20:14 temp\u33.e02s0\u0\u94487028\u93__head_00000E02__ffffffffffffffdd_ffffffffffffffff_0
checking on osd-114
total 1368
-rw-r--r-- 1 ceph ceph 1399392 Nov 20 20:14 temp\u33.e02s0\u0\u94487028\u93__head_00000E02__ffffffffffffffdd_ffffffffffffffff_1
checking on osd-104
total 1368
-rw-r--r-- 1 ceph ceph 1399392 Nov 20 20:14 temp\u33.e02s0\u0\u94487028\u93__head_00000E02__ffffffffffffffdd_ffffffffffffffff_2
checking on osd-91
total 1368
-rw-r--r-- 1 ceph ceph 1399392 Nov 20 20:14 temp\u33.e02s0\u0\u94487028\u93__head_00000E02__ffffffffffffffdd_ffffffffffffffff_3

Does that help?

#9 Updated by David Zafman over 3 years ago

Yes, I just wanted to confirm. So only "temp\u33.e02s0\u0\u93385203\u2__head_00000000__none_ffffffffffffffff_0" is an old hammer one. I've got a pull request that cleans that up on OSD start-up. I'm not sure the ramifications of just removing that file.

https://github.com/ceph/ceph/pull/6976

#10 Updated by Sage Weil over 3 years ago

  • Related to Bug #13381: osd/SnapMapper.cc: 282: FAILED assert(check(oid)) on hammer->jewel upgrade added

#11 Updated by Sage Weil over 3 years ago

  • Status changed from Need Review to Pending Backport

#12 Updated by Loic Dachary over 3 years ago

  • Backport changed from Infernalis to infernalis

#13 Updated by Loic Dachary over 3 years ago

  • Copied to Backport #14494: infernalis: pgs stuck inconsistent after infernalis upgrade added

#14 Updated by Nathan Cutler over 3 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF