Project

General

Profile

Actions
Copy link

Bug #39096

closed

common/Timer.cc: 50: FAILED ceph_assert(thread == __null) in ~SafeTimer

Added by Sage Weil about 5 years ago. Updated almost 2 years ago.

Status:
Can't reproduce
Priority:
High
Assignee:
Brad Hubbard
Category:
MgrClient
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2019-04-03T14:43:57.573 INFO:tasks.ceph.mgr.x.smithi190.stderr:2019-04-03 14:43:57.571 7f293e996d00 -1 mgr init Authentication failed, did you specify a mgr ID with a valid keyring?
2019-04-03T14:43:57.573 INFO:tasks.ceph.mgr.x.smithi190.stderr:Error in initialization: (13) Permission denied
2019-04-03T14:43:57.574 INFO:tasks.ceph.mgr.x.smithi190.stderr:/build/ceph-14.2.0-876-ge76c68f/src/common/Timer.cc: In function 'virtual SafeTimer::~SafeTimer()' thread 7f293e996d00 time 2019-04-03 14:43:57.574091
2019-04-03T14:43:57.574 INFO:tasks.ceph.mgr.x.smithi190.stderr:/build/ceph-14.2.0-876-ge76c68f/src/common/Timer.cc: 50: FAILED ceph_assert(thread == __null)
2019-04-03T14:43:57.578 INFO:tasks.ceph.mgr.x.smithi190.stderr: ceph version 14.2.0-876-ge76c68f (e76c68f79b0309bc601786f1ddf96cc319e43c2f) octopus (dev)
2019-04-03T14:43:57.578 INFO:tasks.ceph.mgr.x.smithi190.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f2935b4f082]
2019-04-03T14:43:57.578 INFO:tasks.ceph.mgr.x.smithi190.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f2935b4f25d]
2019-04-03T14:43:57.579 INFO:tasks.ceph.mgr.x.smithi190.stderr: 3: (()+0x36b33e) [0x7f2935bde33e]
2019-04-03T14:43:57.579 INFO:tasks.ceph.mgr.x.smithi190.stderr: 4: (MgrClient::~MgrClient()+0xfb) [0x598d8b]
2019-04-03T14:43:57.579 INFO:tasks.ceph.mgr.x.smithi190.stderr: 5: (MgrStandby::~MgrStandby()+0x1e8) [0x58e118]
2019-04-03T14:43:57.579 INFO:tasks.ceph.mgr.x.smithi190.stderr: 6: (main()+0x244) [0x4aa3c4]
2019-04-03T14:43:57.579 INFO:tasks.ceph.mgr.x.smithi190.stderr: 7: (__libc_start_main()+0xf0) [0x7f29345a0830]
2019-04-03T14:43:57.579 INFO:tasks.ceph.mgr.x.smithi190.stderr: 8: (_start()+0x29) [0x4ada09]

/a/sage-2019-04-03_02:18:56-rados-wip-sage2-testing-2019-04-02-1625-distro-basic-smithi/3803240

this is the rebuild mondb test, so probably other things going on, but this probably indicates a broken shutdown/error path

Actions
Copy link
 #1

Updated by Sebastian Wagner over 4 years ago

  • Category set to MgrClient
Actions
Copy link
 #2

Updated by Neha Ojha over 4 years ago

  • Assignee set to Brad Hubbard

Brad can you verify if this has been fixed already?

Actions
Copy link
 #3

Updated by Brad Hubbard over 4 years ago

Hi Neha, No, I can't. this is not https://tracker.ceph.com/issues/23352 That was a segfault and this is an assert. I'll see if I can reproduce this issue for a start and see how far we get.

Actions
Copy link
 #4

Updated by Brad Hubbard over 4 years ago

  • Status changed from 12 to Need More Info

The original binaries were built for Xenial and are now gone. We no longer build Xenial binaries by default and I'm not sure how to make this happen? I attempted to reproduce this multiple times by re-building the binaries and running the Bionic binaries but did not see this issue. Please update this tracker with the link to the job if you see this issue.

Actions
Copy link
 #5

Updated by Sage Weil over 4 years ago

  • Status changed from Need More Info to Can't reproduce
Actions
Copy link
 #6

Updated by tao song almost 2 years ago

ceph tell mds.xxx injectargs '--mds_cache_memory_limit 1652177345' assert fail,
common/Timer.cc: 50: FAILED ceph_assert(thread __null) in ~SafeTimer

stack:
(gdb) bt
#0 ceph::__ceph_assert_fail (assertion=assertion@entry=0x7fffe647a6e6 "thread null", file=file@entry=0x7fffe647a660 "/RcVclusters/compile/rpmbuild/BUILD/ceph-12.2.12/src/common/Timer.cc", line=line@entry=50,
func=func@entry=0x7fffe647a9d0 <SafeTimer::~SafeTimer()::__PRETTY_FUNCTION> "SafeTimer::~SafeTimer()") at /usr/src/debug/ceph-12.2.12/src/common/assert.cc:51
#1 0x00007fffe5fe52e3 in SafeTimer::~SafeTimer (this=0x8bb2e0, _in_chrg=<optimized out>) at /usr/src/debug/ceph-12.2.12/src/common/Timer.cc:50
#2 0x00007fffd4530caf in Client::~Client (this=0x8baa40, __in_chrg=<optimized out>) at /usr/src/debug/ceph-12.2.12/src/client/Client.cc:366
#3 0x00007fffd4530f39 in StandaloneClient::~StandaloneClient (this=0x8baa40, __in_chrg=<optimized out>) at /usr/src/debug/ceph-12.2.12/src/client/Client.cc:15145
#4 0x00007fffd44e9710 in shutdown (this=0x7c7fd0) at /usr/src/debug/ceph-12.2.12/src/libcephfs.cc:165
#5 ceph_shutdown (cmount=0x7c7fd0) at /usr/src/debug/ceph-12.2.12/src/libcephfs.cc:358
#6 0x00007fffdc0b4252 in __pyx_pf_6cephfs_9LibCephFS_12shutdown (_pyx_v_self=0x7fffdf48f8d0, _pyx_v_self=0x7fffdf48f8d0) at /usr/src/debug/ceph-12.2.12/build/src/pybind/cephfs/pyrex/cephfs.c:4370
#7 __pyx_pw_6cephfs_9LibCephFS_13shutdown (_pyx_v_self=0x7fffdf48f8d0, unused=<optimized out>) at /usr/src/debug/ceph-12.2.12/build/src/pybind/cephfs/pyrex/cephfs.c:4299
#8 0x00007ffff7af3a7c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#9 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#10 0x00007ffff7a7fa6d in function_call () from /lib64/libpython2.7.so.1.0
#11 0x00007ffff7a5aa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#12 0x00007ffff7aee6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#13 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#14 0x00007ffff7af353c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#15 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#16 0x00007ffff7af353c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#17 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#18 0x00007ffff7af353c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#19 0x00007ffff7af603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#20 0x00007ffff7af6142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#21 0x00007ffff7b0f57f in run_mod () from /lib64/libpython2.7.so.1.0
#22 0x00007ffff7b1073e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
#23 0x00007ffff7b119c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
#24 0x00007ffff7b22b7f in Py_Main () from /lib64/libpython2.7.so.1.0
#25 0x00007ffff6d3f3d5 in __libc_start_main () from /lib64/libc.so.6
#26 0x000000000040066e in _start ()

in Client::shutdown() call SafeTimer::shutdown() ,delete SafeTimerThread *thread and “thread = NULL”，but Client::~Client（）call SafeTimer::~SafeTimer() assert(thread == NULL)

Actions
Copy link

Also available in: Atom PDF