Actions
Bug #44294
closedmds: "elist.h: 91: FAILED ceph_assert(_head.empty())"
Added by Patrick Donnelly about 4 years ago. Updated over 3 years ago.
% Done:
0%
Source:
Development
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
From the LRC testing Octopus:
root@reesi001:~# ceph crash info 2020-02-24T03:31:36.013391Z_9904f466-37f3-43c7-a983-32ebc5a939d8
{
"assert_condition": "_head.empty()",
"assert_file": "/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.1.0-1093-g42bf1cc/rpm/el8/BUILD/ceph-15.1.0-1093-g42bf1cc/src/include/elist.h",
"assert_func": "elist<T>::~elist() [with T = MDSIOContextBase*]",
"assert_line": 91,
"assert_msg": "/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.1.0-1093-g42bf1cc/rpm/el8/BUILD/ceph-15.1.0-1093-g42bf1cc/src/include/elist.h: In function 'elist<T>::~elist() [with T = MDSIOContextBase*]' thread 7f4160f97700 time 2020-02-24T03:31:36.009479+0000\n/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.1.0-1093-g42bf1cc/rpm/el8/BUILD/ceph-15.1.0-1093-g42bf1cc/src/include/elist.h: 91: FAILED ceph_assert(_head.empty())\n",
"assert_thread_name": "MR_Finisher",
"backtrace": [
"(()+0x12dc0) [0x7f416e131dc0]",
"(abort()+0x203) [0x7f416cbdfdd1]",
"(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f416f2fae77]",
"(()+0x279040) [0x7f416f2fb040]",
"(()+0x414ad7) [0x562b74206ad7]",
"(()+0x3a06c) [0x7f416cbf806c]",
"(on_exit()+0) [0x7f416cbf81a0]",
"(()+0x4a7640) [0x562b74299640]",
"(()+0x12dc0) [0x7f416e131dc0]",
"(gsignal()+0x10f) [0x7f416cbf58df]",
"(abort()+0x127) [0x7f416cbdfcf5]",
"(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f416f2fae77]",
"(()+0x279040) [0x7f416f2fb040]",
"(()+0x2cd631) [0x562b740bf631]",
"(MDSContext::complete(int)+0x56) [0x562b74206206]",
"(MDSIOContextBase::complete(int)+0x197) [0x562b74206507]",
"(Finisher::finisher_thread_entry()+0x1a5) [0x7f416f38b5a5]",
"(()+0x82de) [0x7f416e1272de]",
"(clone()+0x43) [0x7f416ccba133]"
],
"ceph_version": "15.1.0-1093-g42bf1cc",
"crash_id": "2020-02-24T03:31:36.013391Z_9904f466-37f3-43c7-a983-32ebc5a939d8",
"entity_name": "mds.reesi002",
"os_id": "centos",
"os_name": "CentOS Linux",
"os_version": "8 (Core)",
"os_version_id": "8",
"process_name": "ceph-mds",
"stack_sig": "cbbfe537c26445d934c542d66875091cdcd9e38f80b5f433e8d97b5853ff604e",
"timestamp": "2020-02-24T03:31:36.013391Z",
"utsname_hostname": "reesi002",
"utsname_machine": "x86_64",
"utsname_release": "4.4.0-116-generic",
"utsname_sysname": "Linux",
"utsname_version": "#140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018"
}
Updated by Patrick Donnelly about 4 years ago
- Related to Bug #44295: mds: MDCache.cc: 6400: FAILED ceph_assert(r == 0 || r == -2) added
Updated by Zheng Yan about 4 years ago
- Status changed from New to Resolved
- Pull request ID set to 33538
resolved, but also introduce new bug. Newest ticket is https://tracker.ceph.com/issues/44680
Updated by Mark Nelson almost 4 years ago
Possibly hit a new manifestation of this while testing ceph.dir.pin.distributed for the IO500 challenge:
2020-07-07T13:59:07.557+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
2020-07-07T13:59:07.559+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
2020-07-07T13:59:07.559+0000 7f7dd17f5600 0 ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev), process ceph-mds, pid 25530
2020-07-07T13:59:07.560+0000 7f7dd17f5600 0 pidfile_write: ignore empty --pid-file
2020-07-07T13:59:07.560+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x107
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x646
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x647
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x648
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x649
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64a
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64b
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64c
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64d
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64e
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64f
2020-07-07T15:27:20.984+0000 7f7dccf8b700 -1 /home/fedora/src/ceph/ceph/src/include/elist.h: In function 'elist<T>::~elist() [with T = MDLockCache*]' thread 7f7dccf8b700 time 2020-07-07T15:27:20.979768+0000
/home/fedora/src/ceph/ceph/src/include/elist.h: 91: FAILED ceph_assert(_head.empty())
ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7f7dd262a774]
2: (()+0x2508ff) [0x7f7dd262a8ff]
3: (std::_Rb_tree<client_t, std::pair<client_t const, Capability>, std::_Select1st<std::pair<client_t const, Capability> >, std::less<client_t>, mempool::pool_allocator<(mempool::pool_index_t)18, std::pair<client_t const, Capability> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<client_t const, Capability> >)+0x15a) [0x562ad7020a4a]
4: (CInode::remove_client_cap(client_t)+0x221) [0x562ad6ffdf01]
5: (CInode::clear_client_caps_after_export()+0x3b) [0x562ad6ffe28b]
6: (Migrator::finish_export_inode_caps(CInode*, int, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >&)+0x534) [0x562ad6f86804]
7: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x282) [0x562ad6f87402]
8: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x41a) [0x562ad6f8759a]
9: (Migrator::export_finish(CDir*)+0x438) [0x562ad6f89988]
10: (Migrator::handle_export_notify_ack(boost::intrusive_ptr<MExportDirNotifyAck const> const&)+0x383) [0x562ad6f8abe3]
11: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x204) [0x562ad6f8b1f4]
12: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5b7) [0x562ad6d9f097]
13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x4f) [0x562ad6d9f68f]
14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x118) [0x562ad6d7b288]
15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f7dd2842848]
16: (DispatchQueue::entry()+0x5ef) [0x7f7dd283ffef]
17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7dd28f7ecd]
18: (()+0x9432) [0x7f7dd1e68432]
19: (clone()+0x43) [0x7f7dd19c29d3]
2020-07-07T15:27:20.986+0000 7f7dccf8b700 -1 *** Caught signal (Aborted) **
in thread 7f7dccf8b700 thread_name:ms_dispatch
ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev)
1: (()+0x14a90) [0x7f7dd1e73a90]
2: (gsignal()+0x145) [0x7f7dd18fda25]
3: (abort()+0x127) [0x7f7dd18e6895]
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x7f7dd262a7be]
5: (()+0x2508ff) [0x7f7dd262a8ff]
6: (std::_Rb_tree<client_t, std::pair<client_t const, Capability>, std::_Select1st<std::pair<client_t const, Capability> >, std::less<client_t>, mempool::pool_allocator<(mempool::pool_index_t)18, std::pair<client_t const, Capability> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<client_t const, Capability> >)+0x15a) [0x562ad7020a4a]
7: (CInode::remove_client_cap(client_t)+0x221) [0x562ad6ffdf01]
8: (CInode::clear_client_caps_after_export()+0x3b) [0x562ad6ffe28b]
9: (Migrator::finish_export_inode_caps(CInode*, int, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >&)+0x534) [0x562ad6f86804]
10: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x282) [0x562ad6f87402]
11: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x41a) [0x562ad6f8759a]
12: (Migrator::export_finish(CDir*)+0x438) [0x562ad6f89988]
13: (Migrator::handle_export_notify_ack(boost::intrusive_ptr<MExportDirNotifyAck const> const&)+0x383) [0x562ad6f8abe3]
14: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x204) [0x562ad6f8b1f4]
15: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5b7) [0x562ad6d9f097]
16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x4f) [0x562ad6d9f68f]
17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x118) [0x562ad6d7b288]
18: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f7dd2842848]
19: (DispatchQueue::entry()+0x5ef) [0x7f7dd283ffef]
20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7dd28f7ecd]
21: (()+0x9432) [0x7f7dd1e68432]
22: (clone()+0x43) [0x7f7dd19c29d3]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-63> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command assert hook 0x562ad93c40f0
-62> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command abort hook 0x562ad93c40f0
-61> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command leak_some_memory hook 0x562ad93c40f0
-60> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perfcounters_dump hook 0x562ad93c40f0
-59> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command 1 hook 0x562ad93c40f0
-58> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perf dump hook 0x562ad93c40f0
-57> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perfcounters_schema hook 0x562ad93c40f0
-56> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perf histogram dump hook 0x562ad93c40f0
-55> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command 2 hook 0x562ad93c40f0
-54> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perf schema hook 0x562ad93c40f0
-53> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perf histogram schema hook 0x562ad93c40f0
-52> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perf reset hook 0x562ad93c40f0
-51> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config show hook 0x562ad93c40f0
-50> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config help hook 0x562ad93c40f0
-49> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config set hook 0x562ad93c40f0
-48> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config unset hook 0x562ad93c40f0
-47> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config get hook 0x562ad93c40f0
-46> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config diff hook 0x562ad93c40f0
-45> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config diff get hook 0x562ad93c40f0
-44> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command injectargs hook 0x562ad93c40f0
-43> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command log flush hook 0x562ad93c40f0
-42> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command log dump hook 0x562ad93c40f0
-41> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command log reopen hook 0x562ad93c40f0
-40> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command dump_mempools hook 0x562ada030068
-39> 2020-07-07T13:59:07.557+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
-38> 2020-07-07T13:59:07.559+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
-37> 2020-07-07T13:59:07.559+0000 7f7dd17f5600 0 ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev), process ceph-mds, pid 25530
-36> 2020-07-07T13:59:07.560+0000 7f7dd17f5600 0 pidfile_write: ignore empty --pid-file
-35> 2020-07-07T13:59:07.560+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
-34> 2020-07-07T13:59:07.569+0000 7f7dd17f5600 1 finished global_init_daemonize
-33> 2020-07-07T13:59:08.283+0000 7f7dccf8b700 4 mgrc handle_mgr_map Got map version 4
-32> 2020-07-07T13:59:08.283+0000 7f7dccf8b700 4 mgrc handle_mgr_map Active mgr is now [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
-31> 2020-07-07T13:59:08.283+0000 7f7dccf8b700 4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
-30> 2020-07-07T13:59:08.284+0000 7f7dccf8b700 4 mgrc handle_mgr_configure stats_period=5
-29> 2020-07-07T13:59:08.284+0000 7f7dccf8b700 4 mgrc handle_mgr_configure updated stats threshold: 5
-28> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x107
-27> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x646
-26> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x647
-25> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x648
-24> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x649
-23> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64a
-22> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64b
-21> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64c
-20> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64d
-19> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64e
-18> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64f
-17> 2020-07-07T14:29:08.284+0000 7f7dccf8b700 4 mgrc ms_handle_reset ms_handle_reset con 0x562ad94d9000
-16> 2020-07-07T14:29:08.284+0000 7f7dccf8b700 4 mgrc reconnect Terminating session with v2:10.0.1.1:6800/18339
-15> 2020-07-07T14:29:08.284+0000 7f7dccf8b700 4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
-14> 2020-07-07T14:29:08.284+0000 7f7dccf8b700 4 mgrc handle_mgr_configure stats_period=5
-13> 2020-07-07T14:44:08.284+0000 7f7dccf8b700 4 mgrc ms_handle_reset ms_handle_reset con 0x562ae12fd400
-12> 2020-07-07T14:44:08.284+0000 7f7dccf8b700 4 mgrc reconnect Terminating session with v2:10.0.1.1:6800/18339
-11> 2020-07-07T14:44:08.284+0000 7f7dccf8b700 4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
-10> 2020-07-07T14:44:08.284+0000 7f7dccf8b700 4 mgrc handle_mgr_configure stats_period=5
-9> 2020-07-07T14:59:08.284+0000 7f7dccf8b700 4 mgrc ms_handle_reset ms_handle_reset con 0x562ad94d9000
-8> 2020-07-07T14:59:08.284+0000 7f7dccf8b700 4 mgrc reconnect Terminating session with v2:10.0.1.1:6800/18339
-7> 2020-07-07T14:59:08.284+0000 7f7dccf8b700 4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
-6> 2020-07-07T14:59:08.284+0000 7f7dccf8b700 4 mgrc handle_mgr_configure stats_period=5
-5> 2020-07-07T15:14:08.285+0000 7f7dccf8b700 4 mgrc ms_handle_reset ms_handle_reset con 0x562ae08b6c00
-4> 2020-07-07T15:14:08.285+0000 7f7dccf8b700 4 mgrc reconnect Terminating session with v2:10.0.1.1:6800/18339
-3> 2020-07-07T15:14:08.285+0000 7f7dccf8b700 4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
-2> 2020-07-07T15:14:08.285+0000 7f7dccf8b700 4 mgrc handle_mgr_configure stats_period=5
-1> 2020-07-07T15:27:20.984+0000 7f7dccf8b700 -1 /home/fedora/src/ceph/ceph/src/include/elist.h: In function 'elist<T>::~elist() [with T = MDLockCache*]' thread 7f7dccf8b700 time 2020-07-07T15:27:20.979768+0000
/home/fedora/src/ceph/ceph/src/include/elist.h: 91: FAILED ceph_assert(_head.empty())
ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7f7dd262a774]
2: (()+0x2508ff) [0x7f7dd262a8ff]
3: (std::_Rb_tree<client_t, std::pair<client_t const, Capability>, std::_Select1st<std::pair<client_t const, Capability> >, std::less<client_t>, mempool::pool_allocator<(mempool::pool_index_t)18, std::pair<client_t const, Capability> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<client_t const, Capability> >)+0x15a) [0x562ad7020a4a]
4: (CInode::remove_client_cap(client_t)+0x221) [0x562ad6ffdf01]
5: (CInode::clear_client_caps_after_export()+0x3b) [0x562ad6ffe28b]
6: (Migrator::finish_export_inode_caps(CInode*, int, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >&)+0x534) [0x562ad6f86804]
7: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x282) [0x562ad6f87402]
8: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x41a) [0x562ad6f8759a]
9: (Migrator::export_finish(CDir*)+0x438) [0x562ad6f89988]
10: (Migrator::handle_export_notify_ack(boost::intrusive_ptr<MExportDirNotifyAck const> const&)+0x383) [0x562ad6f8abe3]
11: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x204) [0x562ad6f8b1f4]
12: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5b7) [0x562ad6d9f097]
13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x4f) [0x562ad6d9f68f]
14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x118) [0x562ad6d7b288]
15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f7dd2842848]
16: (DispatchQueue::entry()+0x5ef) [0x7f7dd283ffef]
17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7dd28f7ecd]
18: (()+0x9432) [0x7f7dd1e68432]
19: (clone()+0x43) [0x7f7dd19c29d3]
0> 2020-07-07T15:27:20.986+0000 7f7dccf8b700 -1 *** Caught signal (Aborted) **
in thread 7f7dccf8b700 thread_name:ms_dispatch
ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev)
1: (()+0x14a90) [0x7f7dd1e73a90]
2: (gsignal()+0x145) [0x7f7dd18fda25]
3: (abort()+0x127) [0x7f7dd18e6895]
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x7f7dd262a7be]
5: (()+0x2508ff) [0x7f7dd262a8ff]
6: (std::_Rb_tree<client_t, std::pair<client_t const, Capability>, std::_Select1st<std::pair<client_t const, Capability> >, std::less<client_t>, mempool::pool_allocator<(mempool::pool_index_t)18, std::pair<client_t const, Capability> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<client_t const, Capability> >)+0x15a) [0x562ad7020a4a]
7: (CInode::remove_client_cap(client_t)+0x221) [0x562ad6ffdf01]
8: (CInode::clear_client_caps_after_export()+0x3b) [0x562ad6ffe28b]
9: (Migrator::finish_export_inode_caps(CInode*, int, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >&)+0x534) [0x562ad6f86804]
10: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x282) [0x562ad6f87402]
11: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x41a) [0x562ad6f8759a]
12: (Migrator::export_finish(CDir*)+0x438) [0x562ad6f89988]
13: (Migrator::handle_export_notify_ack(boost::intrusive_ptr<MExportDirNotifyAck const> const&)+0x383) [0x562ad6f8abe3]
14: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x204) [0x562ad6f8b1f4]
15: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5b7) [0x562ad6d9f097]
16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x4f) [0x562ad6d9f68f]
17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x118) [0x562ad6d7b288]
18: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f7dd2842848]
19: (DispatchQueue::entry()+0x5ef) [0x7f7dd283ffef]
20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7dd28f7ecd]
21: (()+0x9432) [0x7f7dd1e68432]
22: (clone()+0x43) [0x7f7dd19c29d3]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
0/ 0 mds
0/ 0 mds_balancer
0/ 0 mds_locker
0/ 0 mds_log
0/ 0 mds_log_expire
0/ 0 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 0 filer
0/ 1 striper
0/ 0 objecter
0/ 0 rados
0/ 0 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_rwl
0/ 0 journaler
0/ 0 objectcacher
0/ 5 immutable_obj_cache
0/ 0 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
0/ 0 journal
0/ 0 ms
0/ 0 mon
0/ 0 monc
0/ 0 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
1/ 1 reserver
0/ 0 heartbeatmap
0/ 0 perfcounter
0/ 0 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 compressor
5/ 5 bluestore
0/ 0 bluefs
0/ 0 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
5/ 5 prioritycache
0/ 5 test
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
7f7dccf8b700 / ms_dispatch
7f7dd17f5600 / ceph-mds
max_recent 10000
max_new 1000
log_file /tmp/cbt/ceph/log/mds.d.log
--- end dump of recent events ---
Updated by Patrick Donnelly almost 4 years ago
- Status changed from Resolved to Pending Backport
- Backport set to nautilus
This should have been flagged for backport.
Updated by Patrick Donnelly almost 4 years ago
- Copied to Backport #46778: nautilus: mds: "elist.h: 91: FAILED ceph_assert(_head.empty())" added
Updated by Patrick Donnelly almost 4 years ago
- Status changed from Pending Backport to Resolved
Updated by Nathan Cutler over 3 years ago
This issue and #44295 were both fixed by the same PR, https://github.com/ceph/ceph/pull/33538, which was backported to nautilus via #44295.
Actions