Actions
Bug #40724
closedmgr: recursive lock in vstart
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
vstart stuck at:
$ env MDS=1 ../src/vstart.sh -d -b -l -n ** going verbose ** rm -f core* ip 127.0.0.1 port 40331 NOTE: hostname resolves to loopback; remote hosts will not be able to connect. either adjust /etc/hosts, or edit this script to use your machine's real IP. /home/pdonnell/ceph/build/bin/ceph-authtool --create-keyring --gen-key --name=mon. /home/pdonnell/ceph/build/keyring --cap mon 'allow *' creating /home/pdonnell/ceph/build/keyring /home/pdonnell/ceph/build/bin/ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /home/pdonnell/ceph/build/keyring /home/pdonnell/ceph/build/bin/ceph-authtool --gen-key --name=client.fs --cap mon 'allow r' --cap osd 'allow rw tag cephfs data=*' --cap mds 'allow rwp' /home/pdonnell/ceph/build/keyring /home/pdonnell/ceph/build/bin/ceph-authtool --gen-key --name=client.rgw --cap mon 'allow rw' --cap osd 'allow rwx' --cap mgr 'allow rw' /home/pdonnell/ceph/build/keyring /home/pdonnell/ceph/build/bin/monmaptool --create --clobber --addv a [v2:127.0.0.1:40331,v1:127.0.0.1:40332] --addv b [v2:127.0.0.1:40333,v1:127.0.0.1:40334] --addv c [v2:127.0.0.1:40335,v1:127.0.0.1:40336] --print /tmp/ceph_monmap.5728 /home/pdonnell/ceph/build/bin/monmaptool: monmap file /tmp/ceph_monmap.5728 /home/pdonnell/ceph/build/bin/monmaptool: generated fsid f8d12fe4-485f-4784-8edf-add19850b4c6 epoch 0 fsid f8d12fe4-485f-4784-8edf-add19850b4c6 last_changed 2019-07-10T16:29:13.152410-0400 created 2019-07-10T16:29:13.152410-0400 min_mon_release 0 (unknown) 0: [v2:127.0.0.1:40331/0,v1:127.0.0.1:40332/0] mon.a 1: [v2:127.0.0.1:40333/0,v1:127.0.0.1:40334/0] mon.b 2: [v2:127.0.0.1:40335/0,v1:127.0.0.1:40336/0] mon.c /home/pdonnell/ceph/build/bin/monmaptool: writing epoch 0 to /tmp/ceph_monmap.5728 (3 monitors) rm -rf -- /home/pdonnell/ceph/build/dev/mon.a mkdir -p /home/pdonnell/ceph/build/dev/mon.a /home/pdonnell/ceph/build/bin/ceph-mon --mkfs -c /home/pdonnell/ceph/build/ceph.conf -i a --monmap=/tmp/ceph_monmap.5728 --keyring=/home/pdonnell/ceph/build/keyring rm -rf -- /home/pdonnell/ceph/build/dev/mon.b mkdir -p /home/pdonnell/ceph/build/dev/mon.b /home/pdonnell/ceph/build/bin/ceph-mon --mkfs -c /home/pdonnell/ceph/build/ceph.conf -i b --monmap=/tmp/ceph_monmap.5728 --keyring=/home/pdonnell/ceph/build/keyring rm -rf -- /home/pdonnell/ceph/build/dev/mon.c mkdir -p /home/pdonnell/ceph/build/dev/mon.c /home/pdonnell/ceph/build/bin/ceph-mon --mkfs -c /home/pdonnell/ceph/build/ceph.conf -i c --monmap=/tmp/ceph_monmap.5728 --keyring=/home/pdonnell/ceph/build/keyring rm -- /tmp/ceph_monmap.5728 /home/pdonnell/ceph/build/bin/ceph-mon -i a -c /home/pdonnell/ceph/build/ceph.conf /home/pdonnell/ceph/build/bin/ceph-mon -i b -c /home/pdonnell/ceph/build/ceph.conf /home/pdonnell/ceph/build/bin/ceph-mon -i c -c /home/pdonnell/ceph/build/ceph.conf Populating config ... Setting debug configs ... creating /home/pdonnell/ceph/build/dev/mgr.x/keyring /home/pdonnell/ceph/build/bin/ceph -c /home/pdonnell/ceph/build/ceph.conf -k /home/pdonnell/ceph/build/keyring -i /home/pdonnell/ceph/build/dev/mgr.x/keyring auth add mgr.x mon 'allow profile mgr' mds 'allow *' osd 'allow *' added key for mgr.x /home/pdonnell/ceph/build/bin/ceph -c /home/pdonnell/ceph/build/ceph.conf -k /home/pdonnell/ceph/build/keyring config set mgr mgr/dashboard/x/ssl_server_port 41331 --force /home/pdonnell/ceph/build/bin/ceph -c /home/pdonnell/ceph/build/ceph.conf -k /home/pdonnell/ceph/build/keyring config set mgr mgr/restful/x/server_port 42331 --force Starting mgr.x /home/pdonnell/ceph/build/bin/ceph-mgr -i x -c /home/pdonnell/ceph/build/ceph.conf /home/pdonnell/ceph/build/bin/ceph -c /home/pdonnell/ceph/build/ceph.conf -k /home/pdonnell/ceph/build/keyring tell mgr dashboard ac-user-create admin admin administrator
-5> 2019-07-10T16:29:24.554-0400 7f285a4e0d00 1 mgr[py] Loading python module 'ansible' -4> 2019-07-10T16:29:24.554-0400 7f285a4e0d00 1 lockdep using id 40 -3> 2019-07-10T16:29:24.562-0400 7f285a4e0d00 10 mgr[py] Computed sys.path '/home/pdonnell/ceph/src/pybind:/home/pdonnell/ceph/build/lib/cython_modules/lib.2::/usr/lib/python2.7/:/usr/lib/python2.7/plat-x86_64-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload:/home/pdonnell/ceph/src/pybind/mgr:/usr/local/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages' -2> 2019-07-10T16:29:24.718-0400 7f285a4e0d00 0 recursive lock of (5) ceph version v15.0.0-2597-gac64007 (ac640077da858a521d28036f7d3e7ff1889c347f) octopus (dev) 1: (ceph::mutex_debug_detail::mutex_debugging_base::_will_lock(bool)+0x4a) [0x7f28511edc3c] 2: (ceph::shared_mutex_debug::lock()+0x3a) [0x7f285128d16e] 3: (()+0x1582532) [0x7f285112a532] 4: (CRYPTO_mem_ctrl()+0x39) [0x7f284da7c6e9] 5: (()+0x444ee) [0x7f28418ac4ee] 6: (SSL_COMP_get_compression_methods()+0x9) [0x7f28418ae5f9] 7: (SSL_library_init()+0x203) [0x7f28418b3d83] 8: (init_ssl()+0xa2) [0x7f2841ad9c22] 9: (_PyImport_LoadDynamicModule()+0x97) [0x7f284f773ab7] 10: (()+0x15a8d0) [0x7f284f7748d0] 11: (PyImport_ImportModuleLevel()+0x1da) [0x7f284f775b2a] 12: (()+0xb88f8) [0x7f284f6d28f8] 13: (PyObject_Call()+0x43) [0x7f284f72b2b3] 14: (PyEval_CallObjectWithKeywords()+0x47) [0x7f284f801547] 15: (PyEval_EvalFrameEx()+0x38c6) [0x7f284f6c7826] 16: (PyEval_EvalCodeEx()+0x85c) [0x7f284f80211c] 17: (PyEval_EvalCode()+0x19) [0x7f284f6c3de9] 18: (PyImport_ExecCodeModuleEx()+0x94) [0x7f284f772784] 19: (()+0x15915f) [0x7f284f77315f] 20: (()+0x15a8d0) [0x7f284f7748d0] 21: (PyImport_ImportModuleLevel()+0x1da) [0x7f284f775b2a] 22: (()+0xb88f8) [0x7f284f6d28f8] 23: (PyObject_Call()+0x43) [0x7f284f72b2b3] 24: (PyEval_CallObjectWithKeywords()+0x47) [0x7f284f801547] 25: (PyEval_EvalFrameEx()+0x38c6) [0x7f284f6c7826] 26: (PyEval_EvalCodeEx()+0x85c) [0x7f284f80211c] 27: (PyEval_EvalCode()+0x19) [0x7f284f6c3de9] 28: (PyImport_ExecCodeModuleEx()+0x94) [0x7f284f772784] 29: (()+0x15915f) [0x7f284f77315f] 30: (()+0x15a8d0) [0x7f284f7748d0] 31: (PyImport_ImportModuleLevel()+0x1da) [0x7f284f775b2a] 32: (()+0xb88f8) [0x7f284f6d28f8] 33: (PyObject_Call()+0x43) [0x7f284f72b2b3] 34: (PyEval_CallObjectWithKeywords()+0x47) [0x7f284f801547] 35: (PyEval_EvalFrameEx()+0x38c6) [0x7f284f6c7826] 36: (PyEval_EvalCodeEx()+0x85c) [0x7f284f80211c] 37: (PyEval_EvalCode()+0x19) [0x7f284f6c3de9] 38: (PyImport_ExecCodeModuleEx()+0x94) [0x7f284f772784] 39: (()+0x15915f) [0x7f284f77315f] 40: (()+0x15a102) [0x7f284f774102] 41: (()+0x15a8d0) [0x7f284f7748d0] 42: (PyImport_ImportModuleLevel()+0x1da) [0x7f284f775b2a] 43: (()+0xb88f8) [0x7f284f6d28f8] 44: (PyObject_Call()+0x43) [0x7f284f72b2b3] 45: (PyEval_CallObjectWithKeywords()+0x47) [0x7f284f801547] 46: (PyEval_EvalFrameEx()+0x38c6) [0x7f284f6c7826] 47: (PyEval_EvalCodeEx()+0x85c) [0x7f284f80211c] 48: (PyEval_EvalCode()+0x19) [0x7f284f6c3de9] 49: (PyImport_ExecCodeModuleEx()+0x94) [0x7f284f772784] 50: (()+0x15915f) [0x7f284f77315f] 51: (()+0x15a102) [0x7f284f774102] 52: (()+0x15a8d0) [0x7f284f7748d0] 53: (PyImport_ImportModuleLevel()+0x1da) [0x7f284f775b2a] 54: (()+0xb88f8) [0x7f284f6d28f8] 55: (PyObject_Call()+0x43) [0x7f284f72b2b3] 56: (PyEval_CallObjectWithKeywords()+0x47) [0x7f284f801547] 57: (PyEval_EvalFrameEx()+0x38c6) [0x7f284f6c7826] 58: (PyEval_EvalCodeEx()+0x85c) [0x7f284f80211c] 59: (PyEval_EvalCode()+0x19) [0x7f284f6c3de9] 60: (PyImport_ExecCodeModuleEx()+0x94) [0x7f284f772784] 61: (()+0x15915f) [0x7f284f77315f] 62: (()+0x15a102) [0x7f284f774102] 63: (()+0x15a8d0) [0x7f284f7748d0] 64: (PyImport_ImportModuleLevel()+0x62e) [0x7f284f775f7e] 65: (()+0xb88f8) [0x7f284f6d28f8] 66: (PyObject_Call()+0x43) [0x7f284f72b2b3] 67: (PyEval_CallObjectWithKeywords()+0x47) [0x7f284f801547] 68: (PyEval_EvalFrameEx()+0x38c6) [0x7f284f6c7826] 69: (PyEval_EvalCodeEx()+0x85c) [0x7f284f80211c] 70: (PyEval_EvalCode()+0x19) [0x7f284f6c3de9] 71: (PyImport_ExecCodeModuleEx()+0x94) [0x7f284f772784] 72: (()+0x15915f) [0x7f284f77315f] 73: (()+0x15a8d0) [0x7f284f7748d0] 74: (PyImport_ImportModuleLevel()+0x1da) [0x7f284f775b2a] 75: (()+0xb88f8) [0x7f284f6d28f8] 76: (PyObject_Call()+0x43) [0x7f284f72b2b3] 77: (PyEval_CallObjectWithKeywords()+0x47) [0x7f284f801547] 78: (PyEval_EvalFrameEx()+0x38c6) [0x7f284f6c7826] 79: (PyEval_EvalCodeEx()+0x85c) [0x7f284f80211c] 80: (PyEval_EvalCode()+0x19) [0x7f284f6c3de9] 81: (PyImport_ExecCodeModuleEx()+0x94) [0x7f284f772784] 82: (()+0x15915f) [0x7f284f77315f] 83: (()+0x15a102) [0x7f284f774102] 84: (()+0x15a8d0) [0x7f284f7748d0] 85: (PyImport_ImportModuleLevel()+0x1da) [0x7f284f775b2a] 86: (()+0xb88f8) [0x7f284f6d28f8] 87: (PyObject_Call()+0x43) [0x7f284f72b2b3] 88: (PyObject_CallFunction()+0xcf) [0x7f284f72c69f] 89: (PyImport_Import()+0x116) [0x7f284f776536]
With 9aa1eeb37a9c344e34fa513498230cc8765a45b0.
Actions