Bug #39096
closed
common/Timer.cc: 50: FAILED ceph_assert(thread == __null) in ~SafeTimer
Added by Sage Weil about 5 years ago.
Updated almost 2 years ago.
Description
2019-04-03T14:43:57.573 INFO:tasks.ceph.mgr.x.smithi190.stderr:2019-04-03 14:43:57.571 7f293e996d00 -1 mgr init Authentication failed, did you specify a mgr ID with a valid keyring?
2019-04-03T14:43:57.573 INFO:tasks.ceph.mgr.x.smithi190.stderr:Error in initialization: (13) Permission denied
2019-04-03T14:43:57.574 INFO:tasks.ceph.mgr.x.smithi190.stderr:/build/ceph-14.2.0-876-ge76c68f/src/common/Timer.cc: In function 'virtual SafeTimer::~SafeTimer()' thread 7f293e996d00 time 2019-04-03 14:43:57.574091
2019-04-03T14:43:57.574 INFO:tasks.ceph.mgr.x.smithi190.stderr:/build/ceph-14.2.0-876-ge76c68f/src/common/Timer.cc: 50: FAILED ceph_assert(thread == __null)
2019-04-03T14:43:57.578 INFO:tasks.ceph.mgr.x.smithi190.stderr: ceph version 14.2.0-876-ge76c68f (e76c68f79b0309bc601786f1ddf96cc319e43c2f) octopus (dev)
2019-04-03T14:43:57.578 INFO:tasks.ceph.mgr.x.smithi190.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f2935b4f082]
2019-04-03T14:43:57.578 INFO:tasks.ceph.mgr.x.smithi190.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f2935b4f25d]
2019-04-03T14:43:57.579 INFO:tasks.ceph.mgr.x.smithi190.stderr: 3: (()+0x36b33e) [0x7f2935bde33e]
2019-04-03T14:43:57.579 INFO:tasks.ceph.mgr.x.smithi190.stderr: 4: (MgrClient::~MgrClient()+0xfb) [0x598d8b]
2019-04-03T14:43:57.579 INFO:tasks.ceph.mgr.x.smithi190.stderr: 5: (MgrStandby::~MgrStandby()+0x1e8) [0x58e118]
2019-04-03T14:43:57.579 INFO:tasks.ceph.mgr.x.smithi190.stderr: 6: (main()+0x244) [0x4aa3c4]
2019-04-03T14:43:57.579 INFO:tasks.ceph.mgr.x.smithi190.stderr: 7: (__libc_start_main()+0xf0) [0x7f29345a0830]
2019-04-03T14:43:57.579 INFO:tasks.ceph.mgr.x.smithi190.stderr: 8: (_start()+0x29) [0x4ada09]
/a/sage-2019-04-03_02:18:56-rados-wip-sage2-testing-2019-04-02-1625-distro-basic-smithi/3803240
this is the rebuild mondb test, so probably other things going on, but this probably indicates a broken shutdown/error path
- Category set to MgrClient
- Assignee set to Brad Hubbard
Brad can you verify if this has been fixed already?
Hi Neha, No, I can't. this is not https://tracker.ceph.com/issues/23352 That was a segfault and this is an assert. I'll see if I can reproduce this issue for a start and see how far we get.
- Status changed from 12 to Need More Info
The original binaries were built for Xenial and are now gone. We no longer build Xenial binaries by default and I'm not sure how to make this happen? I attempted to reproduce this multiple times by re-building the binaries and running the Bionic binaries but did not see this issue. Please update this tracker with the link to the job if you see this issue.
- Status changed from Need More Info to Can't reproduce
ceph tell mds.xxx injectargs '--mds_cache_memory_limit 1652177345' assert fail,
common/Timer.cc: 50: FAILED ceph_assert(thread __null) in ~SafeTimer
stack:
(gdb) bt
#0 ceph::__ceph_assert_fail (assertion=assertion@entry=0x7fffe647a6e6 "thread null", file=file@entry=0x7fffe647a660 "/RcVclusters/compile/rpmbuild/BUILD/ceph-12.2.12/src/common/Timer.cc", line=line@entry=50,
func=func@entry=0x7fffe647a9d0 <SafeTimer::~SafeTimer()::__PRETTY_FUNCTION> "SafeTimer::~SafeTimer()") at /usr/src/debug/ceph-12.2.12/src/common/assert.cc:51
#1 0x00007fffe5fe52e3 in SafeTimer::~SafeTimer (this=0x8bb2e0, _in_chrg=<optimized out>) at /usr/src/debug/ceph-12.2.12/src/common/Timer.cc:50
#2 0x00007fffd4530caf in Client::~Client (this=0x8baa40, __in_chrg=<optimized out>) at /usr/src/debug/ceph-12.2.12/src/client/Client.cc:366
#3 0x00007fffd4530f39 in StandaloneClient::~StandaloneClient (this=0x8baa40, __in_chrg=<optimized out>) at /usr/src/debug/ceph-12.2.12/src/client/Client.cc:15145
#4 0x00007fffd44e9710 in shutdown (this=0x7c7fd0) at /usr/src/debug/ceph-12.2.12/src/libcephfs.cc:165
#5 ceph_shutdown (cmount=0x7c7fd0) at /usr/src/debug/ceph-12.2.12/src/libcephfs.cc:358
#6 0x00007fffdc0b4252 in __pyx_pf_6cephfs_9LibCephFS_12shutdown (_pyx_v_self=0x7fffdf48f8d0, _pyx_v_self=0x7fffdf48f8d0) at /usr/src/debug/ceph-12.2.12/build/src/pybind/cephfs/pyrex/cephfs.c:4370
#7 __pyx_pw_6cephfs_9LibCephFS_13shutdown (_pyx_v_self=0x7fffdf48f8d0, unused=<optimized out>) at /usr/src/debug/ceph-12.2.12/build/src/pybind/cephfs/pyrex/cephfs.c:4299
#8 0x00007ffff7af3a7c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#9 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#10 0x00007ffff7a7fa6d in function_call () from /lib64/libpython2.7.so.1.0
#11 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#12 0x00007ffff7aee6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#13 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#14 0x00007ffff7af353c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#15 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#16 0x00007ffff7af353c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#17 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#18 0x00007ffff7af353c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#19 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#20 0x00007ffff7af6142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#21 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0
#22 0x00007ffff7b1073e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
#23 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
#24 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0
#25 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6
#26 0x000000000040066e in _start ()
in Client::shutdown() call SafeTimer::shutdown() ,delete SafeTimerThread *thread and “thread = NULL”，but Client::~Client（）call SafeTimer::~SafeTimer() assert(thread == NULL)
Also available in: Atom
PDF