Project

General

Profile

Bug #2324

osd: assert("q.empty()") failed in OpSequencer destructor

Added by Josh Durgin almost 12 years ago. Updated almost 12 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
OSD
Target version:
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This is consistently reproducible with 2 osds started by vstart on vit, but only happens intermittently with 1 osd.
test_librbd triggers this on master and wip-discard. It might be related to cleanup of watchers during pg removal, since just removing a pool doesn't trigger it.
OSD log is attached for a case when there was only one.

 ceph version 0.45-247-ge09250c (commit:e09250c0f27ac085760bd6b9ff1ec3ae1c38e08f)
 1: ./ceph-osd() [0x730db9]
 2: (()+0xef60) [0x7f8e96a08f60]
 3: (gsignal()+0x35) [0x7f8e94fe8165]
 4: (abort()+0x180) [0x7f8e94feaf70]
 5: (__assert_fail()+0xf1) [0x7f8e94fe12b1]
 6: (FileStore::OpSequencer::~OpSequencer()+0xf7) [0x7fea47]
 7: (PG::~PG()+0xce) [0x59264e]
 8: (ReplicatedPG::~ReplicatedPG()+0x1ac) [0x59331c]
 9: (OSD::_remove_pg(PG*)+0x16eb) [0x5d039b]
 10: (ThreadPool::worker()+0xb38) [0x698ed8]
 11: (ThreadPool::WorkThread::entry()+0xd) [0x5effed]
 12: (()+0x68ba) [0x7f8e96a008ba]
 13: (clone()+0x6d) [0x7f8e9508502d]

PG dump is:

dumped all in format plain
version 17
last_osdmap_epoch 15
last_pg_scan 15
full_ratio 0.95
nearfull_ratio 0.85
pg_stat    objects    mip    degr    unf    bytes    log    disklog    state    state_stamp    v    reported    up    acting    last_scrub    scrub_stamp
6.1    0    0    0    0    0    0    0    creating    0.000000    0'0    0'0    []    []    0'0    0.000000
2.5    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.863963    0'0    4'4    [0]    [0]    0'0    0.000000
1.6    1    0    1    0    205    116    116    active+degraded    2012-04-20 15:22:41.865858    5'1    4'5    [0]    [0]    0'0    0.000000
0.7    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.878861    0'0    4'4    [0]    [0]    0'0    0.000000
6.0    0    0    0    0    0    0    0    creating    0.000000    0'0    0'0    []    []    0'0    0.000000
2.4    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.879009    0'0    4'4    [0]    [0]    0'0    0.000000
1.7    5    0    5    0    1395    585    585    active+degraded    2012-04-20 15:22:41.879148    5'5    4'9    [0]    [0]    0'0    0.000000
0.6    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.879376    0'0    4'4    [0]    [0]    0'0    0.000000
6.3    0    0    0    0    0    0    0    creating    0.000000    0'0    0'0    []    []    0'0    0.000000
2.7    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.879516    0'0    4'4    [0]    [0]    0'0    0.000000
1.4    5    0    5    0    1724    700    700    active+degraded    2012-04-20 15:22:41.879653    5'6    4'10    [0]    [0]    0'0    0.000000
0.5    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.879867    0'0    4'4    [0]    [0]    0'0    0.000000
6.2    0    0    0    0    0    0    0    creating    0.000000    0'0    0'0    []    []    0'0    0.000000
2.6    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.880011    0'0    4'4    [0]    [0]    0'0    0.000000
1.5    3    0    3    0    451    349    349    active+degraded    2012-04-20 15:22:41.881206    5'3    4'7    [0]    [0]    0'0    0.000000
0.4    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.881400    0'0    4'4    [0]    [0]    0'0    0.000000
6.5    0    0    0    0    0    0    0    creating    0.000000    0'0    0'0    []    []    0'0    0.000000
2.1    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.881554    0'0    4'4    [0]    [0]    0'0    0.000000
1.2    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.881708    0'0    4'4    [0]    [0]    0'0    0.000000
0.3    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.881880    0'0    4'4    [0]    [0]    0'0    0.000000
6.4    0    0    0    0    0    0    0    creating    0.000000    0'0    0'0    []    []    0'0    0.000000
2.0    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.882028    0'0    4'4    [0]    [0]    0'0    0.000000
1.3    3    0    3    0    4135    352    352    active+degraded    2012-04-20 15:22:41.882178    5'3    4'7    [0]    [0]    0'0    0.000000
0.2    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.882404    0'0    4'4    [0]    [0]    0'0    0.000000
6.7    0    0    0    0    0    0    0    creating    0.000000    0'0    0'0    []    []    0'0    0.000000
2.3    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.882554    0'0    4'4    [0]    [0]    0'0    0.000000
1.0    4    0    4    0    820    464    464    active+degraded    2012-04-20 15:22:41.882703    5'4    4'8    [0]    [0]    0'0    0.000000
0.1    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.882913    0'0    4'4    [0]    [0]    0'0    0.000000
6.6    0    0    0    0    0    0    0    creating    0.000000    0'0    0'0    []    []    0'0    0.000000
2.2    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.883067    0'0    4'4    [0]    [0]    0'0    0.000000
1.1    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.883212    0'0    4'4    [0]    [0]    0'0    0.000000
0.0    0    0    0    0    0    0    0    active+degraded    2012-04-20 15:22:41.883367    0'0    4'4    [0]    [0]    0'0    0.000000
pool 0    0    0    0    0    0    0    0
pool 1    21    0    21    0    8730    2566    2566
pool 2    0    0    0    0    0    0    0
pool 6    0    0    0    0    0    0    0
 sum    21    0    21    0    8730    2566    2566
osdstat    kbused    kbavail    kb    hb in    hb out
0    218364176    639479708    903751800    []    []
 sum    218364176    639479708    903751800

osd.0.log View (4.24 MB) Josh Durgin, 04/20/2012 03:56 PM

Associated revisions

Revision 888a082f (diff)
Added by Samuel Just almost 12 years ago

FileJournal: don't wait flusher until completions are queued

Fixes: #2324
Signed-off-by: Samuel Just <>

History

#1 Updated by Sage Weil almost 12 years ago

  • Status changed from New to Resolved

Also available in: Atom PDF