Actions
Bug #5257
closedCeph OSD bobtail->cuttlefish upgrade goes backward in time with osdmap
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
Monitor
Target version:
-
% Done:
0%
Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I upgraded ceph on one of my boxes (12 osds) and the osds came up, printing "183140 PGs are upgrading". During that time, the cluster has those osds as up, the pgs as degraded/not recovering and a bunch of "slow requests" are getting logged. Based on the numbers I'm seeing, the upgrade is going to take quite a while.
debug-osd=10 has one of those osds as:
2013-06-05 18:37:50.120126 7fb7f8907780 1 osd.50 pg_epoch: 183140 pg[3.1212( v 183140'66684 (180299'65683,183140'66684] local-les=181898 n=16706 ec=14 les/c 181898/181952 181875/181875/179540) [] r=0 lpr=0 pi=155667-181874/107 lcod 0'0 mlcod 0'0 inactive] Updating snap_mapper from main collection, 14347 objects done 2013-06-05 18:37:50.274509 7fb7f8907780 1 osd.50 pg_epoch: 183140 pg[3.1212( v 183140'66684 (180299'65683,183140'66684] local-les=181898 n=16706 ec=14 les/c 181898/181952 181875/181875/179540) [] r=0 lpr=0 pi=155667-181874/107 lcod 0'0 mlcod 0'0 inactive] Updating snap_mapper from main collection, 14411 objects done 2013-06-05 18:37:50.281523 7fb7f8907780 1 osd.50 pg_epoch: 183140 pg[3.1212( v 183140'66684 (180299'65683,183140'66684] local-les=181898 n=16706 ec=14 les/c 181898/181952 181875/181875/179540) [] r=0 lpr=0 pi=155667-181874/107 lcod 0'0 mlcod 0'0 inactive] Updating snap_mapper from main collection, 14475 objects done 2013-06-05 18:37:50.288334 7fb7f8907780 1 osd.50 pg_epoch: 183140 pg[3.1212( v 183140'66684 (180299'65683,183140'66684] local-les=181898 n=16706 ec=14 les/c 181898/181952 181875/181875/179540) [] r=0 lpr=0 pi=155667-181874/107 lcod 0'0 mlcod 0'0 inactive] Updating snap_mapper from main collection, 14539 objects done 2013-06-05 18:37:50.295061 7fb7f8907780 1 osd.50 pg_epoch: 183140 pg[3.1212( v 183140'66684 (180299'65683,183140'66684] local-les=181898 n=16706 ec=14 les/c 181898/181952 181875/181875/179540) [] r=0 lpr=0 pi=155667-181874/107 lcod 0'0 mlcod 0'0 inactive] Updating snap_mapper from main collection, 14603 objects done 2013-06-05 18:37:50.301661 7fb7f8907780 1 osd.50 pg_epoch: 183140 pg[3.1212( v 183140'66684 (180299'65683,183140'66684] local-les=181898 n=16706 ec=14 les/c 181898/181952 181875/181875/179540) [] r=0 lpr=0 pi=155667-181874/107 lcod 0'0 mlcod 0'0 inactive] Updating snap_mapper from main collection, 14646 objects done 2013-06-05 18:37:50.448612 7fb7f8907780 1 osd.50 pg_epoch: 183140 pg[3.1212( v 183140'66684 (180299'65683,183140'66684] local-les=181898 n=16706 ec=14 les/c 181898/181952 181875/181875/179540) [] r=0 lpr=0 pi=155667-181874/107 lcod 0'0 mlcod 0'0 inactive] Updating snap_mapper from main collection, 14710 objects done 2013-06-05 18:37:50.452574 7fb7f8907780 1 osd.50 pg_epoch: 183140 pg[3.1212( v 183140'66684 (180299'65683,183140'66684] local-les=181898 n=16706 ec=14 les/c 181898/181952 181875/181875/179540) [] r=0 lpr=0 pi=155667-181874/107 lcod 0'0 mlcod 0'0 inactive] Updating snap_mapper from main collection, 14774 objects done 2013-06-05 18:37:50.456327 7fb7f8907780 1 osd.50 pg_epoch: 183140 pg[3.1212( v 183140'66684 (180299'65683,183140'66684] local-les=181898 n=16706 ec=14 les/c 181898/181952 181875/181875/179540) [] r=0 lpr=0 pi=155667-181874/107 lcod 0'0 mlcod 0'0 inactive] Updating snap_mapper from main collection, 14838 objects done 2013-06-05 18:37:50.460033 7fb7f8907780 1 osd.50 pg_epoch: 183140 pg[3.1212( v 183140'66684 (180299'65683,183140'66684] local-les=181898 n=16706 ec=14 les/c 181898/181952 181875/181875/179540) [] r=0 lpr=0 pi=155667-181874/107 lcod 0'0 mlcod 0'0 inactive] Updating snap_mapper from main collection, 14902 objects done
Actions