Project

General

Profile

Bug #2135

cephtool: osdc/Objecter.cc: 375: FAILED assert(initialized)

Added by Sage Weil about 12 years ago. Updated about 12 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
librados
Target version:
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2012-03-03T17:10:01.609 DEBUG:teuthology.orchestra.run:Running: 'LD_LIBRARY_PRELOAD=/tmp/cephtest/binary/usr/local/lib /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/binary/usr/local/bin/ceph -k /tmp/cephtest/ceph.keyring -c /tmp/cephtest/ceph.conf --concise osd out 3'
2012-03-03T17:10:04.039 INFO:teuthology.task.rados.rados.0.err:osdc/Objecter.cc: In function 'void Objecter::handle_osd_map(MOSDMap*)' thread 7ffe2b3d6700 time 2012-03-03 17:10:03.234532
2012-03-03T17:10:04.040 INFO:teuthology.task.rados.rados.0.err:osdc/Objecter.cc: 375: FAILED assert(initialized)
2012-03-03T17:10:04.152 INFO:teuthology.task.rados.rados.0.err: ceph version 0.42.2-211-g3a83517 (commit:3a8351725667d6cc363164a3c280246acf53ccd0)
2012-03-03T17:10:04.152 INFO:teuthology.task.rados.rados.0.err: 1: (Objecter::handle_osd_map(MOSDMap*)+0x274a) [0x7ffe2dd18e0a]
2012-03-03T17:10:04.152 INFO:teuthology.task.rados.rados.0.err: 2: (librados::RadosClient::_dispatch(Message*)+0xdc) [0x7ffe2dce41ec]
2012-03-03T17:10:04.152 INFO:teuthology.task.rados.rados.0.err: 3: (librados::RadosClient::ms_dispatch(Message*)+0x263) [0x7ffe2dce44e3]
2012-03-03T17:10:04.153 INFO:teuthology.task.rados.rados.0.err: 4: (SimpleMessenger::dispatch_entry()+0xa9a) [0x7ffe2de55bea]
2012-03-03T17:10:04.153 INFO:teuthology.task.rados.rados.0.err: 5: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x7ffe2dcf22cc]
2012-03-03T17:10:04.153 INFO:teuthology.task.rados.rados.0.err: 6: (Thread::_entry_func(void*)+0x12) [0x7ffe2ded4582]
2012-03-03T17:10:04.153 INFO:teuthology.task.rados.rados.0.err: 7: (()+0x7971) [0x7ffe2d9cb971]
2012-03-03T17:10:04.153 INFO:teuthology.task.rados.rados.0.err: 8: (clone()+0x6d) [0x7ffe2d20b92d]
2012-03-03T17:10:04.154 INFO:teuthology.task.rados.rados.0.err: ceph version 0.42.2-211-g3a83517 (commit:3a8351725667d6cc363164a3c280246acf53ccd0)
2012-03-03T17:10:04.154 INFO:teuthology.task.rados.rados.0.err: 1: (Objecter::handle_osd_map(MOSDMap*)+0x274a) [0x7ffe2dd18e0a]
2012-03-03T17:10:04.154 INFO:teuthology.task.rados.rados.0.err: 2: (librados::RadosClient::_dispatch(Message*)+0xdc) [0x7ffe2dce41ec]
2012-03-03T17:10:04.154 INFO:teuthology.task.rados.rados.0.err: 3: (librados::RadosClient::ms_dispatch(Message*)+0x263) [0x7ffe2dce44e3]
2012-03-03T17:10:04.154 INFO:teuthology.task.rados.rados.0.err: 4: (SimpleMessenger::dispatch_entry()+0xa9a) [0x7ffe2de55bea]
2012-03-03T17:10:04.154 INFO:teuthology.task.rados.rados.0.err: 5: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x7ffe2dcf22cc]
2012-03-03T17:10:04.155 INFO:teuthology.task.rados.rados.0.err: 6: (Thread::_entry_func(void*)+0x12) [0x7ffe2ded4582]
2012-03-03T17:10:04.155 INFO:teuthology.task.rados.rados.0.err: 7: (()+0x7971) [0x7ffe2d9cb971]
2012-03-03T17:10:04.155 INFO:teuthology.task.rados.rados.0.err: 8: (clone()+0x6d) [0x7ffe2d20b92d]
2012-03-03T17:10:04.239 INFO:teuthology.task.rados.rados.0.err:terminate called after throwing an instance of 'ceph::FailedAssertion'

ubuntu@teuthology:/a/nightly_coverage_2012-03-03-b/14937$

Associated revisions

Revision 702f09ea (diff)
Added by Sage Weil about 12 years ago

librados: close narrow shutdown race

timer.shutdown() will drop and retake the lock, so set DISCONNECTED first
to avoid a message slipping in and reaching the objecter like so:

INFO:teuthology.task.rados.rados.0.err:osdc/Objecter.cc: In function 'void Objecter::handle_osd_op_reply(MOSDOpReply*)' thread 7f0bc2b1b700 time 2012-03-03 18:35:25.302135
INFO:teuthology.task.rados.rados.0.err:osdc/Objecter.cc: 1151: FAILED assert(initialized)
INFO:teuthology.task.rados.rados.0.err: ceph version 0.43-46-g2e57997 (commit:2e57997894944696fcc737aae9b57e30b6bb5bdc)
INFO:teuthology.task.rados.rados.0.err: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xb3) [0x7f0bc59bd66f]
INFO:teuthology.task.rados.rados.0.err: 2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x82) [0x7f0bc58e885e]
INFO:teuthology.task.rados.rados.0.err: 3: (librados::RadosClient::_dispatch(Message*)+0x66) [0x7f0bc58a2674]
INFO:teuthology.task.rados.rados.0.err: 4: (librados::RadosClient::ms_dispatch(Message*)+0x130) [0x7f0bc58a246e]
INFO:teuthology.task.rados.rados.0.err: 5: (Messenger::ms_deliver_dispatch(Message*)+0x8b) [0x7f0bc5a4e859]
INFO:teuthology.task.rados.rados.0.err: 6: (SimpleMessenger::dispatch_entry()+0x7c2) [0x7f0bc5a377fc]
INFO:teuthology.task.rados.rados.0.err: 7: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x7f0bc58b5512]
INFO:teuthology.task.rados.rados.0.err: 8: (Thread::_entry_func(void*)+0x23) [0x7f0bc5ac4c75]
INFO:teuthology.task.rados.rados.0.err: 9: (()+0x7971) [0x7f0bc5110971]
INFO:teuthology.task.rados.rados.0.err: 10: (clone()+0x6d) [0x7f0bc495092d]

Fixes: #2135
Signed-off-by: Sage Weil <>
Reviewed-by: Josh Durgin <>

History

#1 Updated by Sage Weil about 12 years ago

  • Status changed from New to 4
  • Assignee set to Sage Weil

librados shutdown race

#2 Updated by Sage Weil about 12 years ago

  • Category set to librados

#3 Updated by Sage Weil about 12 years ago

  • Status changed from 4 to Resolved

Also available in: Atom PDF