Actions
Bug #2324
closedosd: assert("q.empty()") failed in OpSequencer destructor
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
This is consistently reproducible with 2 osds started by vstart on vit, but only happens intermittently with 1 osd.
test_librbd triggers this on master and wip-discard. It might be related to cleanup of watchers during pg removal, since just removing a pool doesn't trigger it.
OSD log is attached for a case when there was only one.
ceph version 0.45-247-ge09250c (commit:e09250c0f27ac085760bd6b9ff1ec3ae1c38e08f) 1: ./ceph-osd() [0x730db9] 2: (()+0xef60) [0x7f8e96a08f60] 3: (gsignal()+0x35) [0x7f8e94fe8165] 4: (abort()+0x180) [0x7f8e94feaf70] 5: (__assert_fail()+0xf1) [0x7f8e94fe12b1] 6: (FileStore::OpSequencer::~OpSequencer()+0xf7) [0x7fea47] 7: (PG::~PG()+0xce) [0x59264e] 8: (ReplicatedPG::~ReplicatedPG()+0x1ac) [0x59331c] 9: (OSD::_remove_pg(PG*)+0x16eb) [0x5d039b] 10: (ThreadPool::worker()+0xb38) [0x698ed8] 11: (ThreadPool::WorkThread::entry()+0xd) [0x5effed] 12: (()+0x68ba) [0x7f8e96a008ba] 13: (clone()+0x6d) [0x7f8e9508502d]
PG dump is:
dumped all in format plain version 17 last_osdmap_epoch 15 last_pg_scan 15 full_ratio 0.95 nearfull_ratio 0.85 pg_stat objects mip degr unf bytes log disklog state state_stamp v reported up acting last_scrub scrub_stamp 6.1 0 0 0 0 0 0 0 creating 0.000000 0'0 0'0 [] [] 0'0 0.000000 2.5 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.863963 0'0 4'4 [0] [0] 0'0 0.000000 1.6 1 0 1 0 205 116 116 active+degraded 2012-04-20 15:22:41.865858 5'1 4'5 [0] [0] 0'0 0.000000 0.7 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.878861 0'0 4'4 [0] [0] 0'0 0.000000 6.0 0 0 0 0 0 0 0 creating 0.000000 0'0 0'0 [] [] 0'0 0.000000 2.4 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.879009 0'0 4'4 [0] [0] 0'0 0.000000 1.7 5 0 5 0 1395 585 585 active+degraded 2012-04-20 15:22:41.879148 5'5 4'9 [0] [0] 0'0 0.000000 0.6 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.879376 0'0 4'4 [0] [0] 0'0 0.000000 6.3 0 0 0 0 0 0 0 creating 0.000000 0'0 0'0 [] [] 0'0 0.000000 2.7 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.879516 0'0 4'4 [0] [0] 0'0 0.000000 1.4 5 0 5 0 1724 700 700 active+degraded 2012-04-20 15:22:41.879653 5'6 4'10 [0] [0] 0'0 0.000000 0.5 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.879867 0'0 4'4 [0] [0] 0'0 0.000000 6.2 0 0 0 0 0 0 0 creating 0.000000 0'0 0'0 [] [] 0'0 0.000000 2.6 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.880011 0'0 4'4 [0] [0] 0'0 0.000000 1.5 3 0 3 0 451 349 349 active+degraded 2012-04-20 15:22:41.881206 5'3 4'7 [0] [0] 0'0 0.000000 0.4 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.881400 0'0 4'4 [0] [0] 0'0 0.000000 6.5 0 0 0 0 0 0 0 creating 0.000000 0'0 0'0 [] [] 0'0 0.000000 2.1 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.881554 0'0 4'4 [0] [0] 0'0 0.000000 1.2 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.881708 0'0 4'4 [0] [0] 0'0 0.000000 0.3 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.881880 0'0 4'4 [0] [0] 0'0 0.000000 6.4 0 0 0 0 0 0 0 creating 0.000000 0'0 0'0 [] [] 0'0 0.000000 2.0 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.882028 0'0 4'4 [0] [0] 0'0 0.000000 1.3 3 0 3 0 4135 352 352 active+degraded 2012-04-20 15:22:41.882178 5'3 4'7 [0] [0] 0'0 0.000000 0.2 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.882404 0'0 4'4 [0] [0] 0'0 0.000000 6.7 0 0 0 0 0 0 0 creating 0.000000 0'0 0'0 [] [] 0'0 0.000000 2.3 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.882554 0'0 4'4 [0] [0] 0'0 0.000000 1.0 4 0 4 0 820 464 464 active+degraded 2012-04-20 15:22:41.882703 5'4 4'8 [0] [0] 0'0 0.000000 0.1 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.882913 0'0 4'4 [0] [0] 0'0 0.000000 6.6 0 0 0 0 0 0 0 creating 0.000000 0'0 0'0 [] [] 0'0 0.000000 2.2 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.883067 0'0 4'4 [0] [0] 0'0 0.000000 1.1 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.883212 0'0 4'4 [0] [0] 0'0 0.000000 0.0 0 0 0 0 0 0 0 active+degraded 2012-04-20 15:22:41.883367 0'0 4'4 [0] [0] 0'0 0.000000 pool 0 0 0 0 0 0 0 0 pool 1 21 0 21 0 8730 2566 2566 pool 2 0 0 0 0 0 0 0 pool 6 0 0 0 0 0 0 0 sum 21 0 21 0 8730 2566 2566 osdstat kbused kbavail kb hb in hb out 0 218364176 639479708 903751800 [] [] sum 218364176 639479708 903751800
Files
Updated by Sage Weil about 12 years ago
- Status changed from New to Resolved
sam fixed this in 888a082f23974b1f7a63f302e29a326182e7dc41
Actions