https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2020-02-25T18:45:40Z
Ceph
CephFS - Bug #44294: mds: "elist.h: 91: FAILED ceph_assert(_head.empty())"
https://tracker.ceph.com/issues/44294?journal_id=159557
2020-02-25T18:45:40Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-6 priority-high2 closed" href="/issues/44295">Bug #44295</a>: mds: MDCache.cc: 6400: FAILED ceph_assert(r == 0 || r == -2)</i> added</li></ul>
CephFS - Bug #44294: mds: "elist.h: 91: FAILED ceph_assert(_head.empty())"
https://tracker.ceph.com/issues/44294?journal_id=161468
2020-03-23T12:13:47Z
Zheng Yan
ukernel@gmail.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Resolved</i></li><li><strong>Pull request ID</strong> set to <i>33538</i></li></ul><p>resolved, but also introduce new bug. Newest ticket is <a class="external" href="https://tracker.ceph.com/issues/44680">https://tracker.ceph.com/issues/44680</a></p>
CephFS - Bug #44294: mds: "elist.h: 91: FAILED ceph_assert(_head.empty())"
https://tracker.ceph.com/issues/44294?journal_id=169910
2020-07-07T15:35:42Z
Mark Nelson
mark.a.nelson@gmail.com
<ul></ul><p>Possibly hit a new manifestation of this while testing ceph.dir.pin.distributed for the IO500 challenge:</p>
<pre>
2020-07-07T13:59:07.557+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
2020-07-07T13:59:07.559+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
2020-07-07T13:59:07.559+0000 7f7dd17f5600 0 ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev), process ceph-mds, pid 25530
2020-07-07T13:59:07.560+0000 7f7dd17f5600 0 pidfile_write: ignore empty --pid-file
2020-07-07T13:59:07.560+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x107
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x646
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x647
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x648
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x649
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64a
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64b
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64c
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64d
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64e
2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64f
2020-07-07T15:27:20.984+0000 7f7dccf8b700 -1 /home/fedora/src/ceph/ceph/src/include/elist.h: In function 'elist<T>::~elist() [with T = MDLockCache*]' thread 7f7dccf8b700 time 2020-07-07T15:27:20.979768+0000
/home/fedora/src/ceph/ceph/src/include/elist.h: 91: FAILED ceph_assert(_head.empty())
ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7f7dd262a774]
2: (()+0x2508ff) [0x7f7dd262a8ff]
3: (std::_Rb_tree<client_t, std::pair<client_t const, Capability>, std::_Select1st<std::pair<client_t const, Capability> >, std::less<client_t>, mempool::pool_allocator<(mempool::pool_index_t)18, std::pair<client_t const, Capability> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<client_t const, Capability> >)+0x15a) [0x562ad7020a4a]
4: (CInode::remove_client_cap(client_t)+0x221) [0x562ad6ffdf01]
5: (CInode::clear_client_caps_after_export()+0x3b) [0x562ad6ffe28b]
6: (Migrator::finish_export_inode_caps(CInode*, int, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >&)+0x534) [0x562ad6f86804]
7: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x282) [0x562ad6f87402]
8: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x41a) [0x562ad6f8759a]
9: (Migrator::export_finish(CDir*)+0x438) [0x562ad6f89988]
10: (Migrator::handle_export_notify_ack(boost::intrusive_ptr<MExportDirNotifyAck const> const&)+0x383) [0x562ad6f8abe3]
11: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x204) [0x562ad6f8b1f4]
12: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5b7) [0x562ad6d9f097]
13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x4f) [0x562ad6d9f68f]
14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x118) [0x562ad6d7b288]
15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f7dd2842848]
16: (DispatchQueue::entry()+0x5ef) [0x7f7dd283ffef]
17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7dd28f7ecd]
18: (()+0x9432) [0x7f7dd1e68432]
19: (clone()+0x43) [0x7f7dd19c29d3]
2020-07-07T15:27:20.986+0000 7f7dccf8b700 -1 *** Caught signal (Aborted) **
in thread 7f7dccf8b700 thread_name:ms_dispatch
ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev)
1: (()+0x14a90) [0x7f7dd1e73a90]
2: (gsignal()+0x145) [0x7f7dd18fda25]
3: (abort()+0x127) [0x7f7dd18e6895]
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x7f7dd262a7be]
5: (()+0x2508ff) [0x7f7dd262a8ff]
6: (std::_Rb_tree<client_t, std::pair<client_t const, Capability>, std::_Select1st<std::pair<client_t const, Capability> >, std::less<client_t>, mempool::pool_allocator<(mempool::pool_index_t)18, std::pair<client_t const, Capability> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<client_t const, Capability> >)+0x15a) [0x562ad7020a4a]
7: (CInode::remove_client_cap(client_t)+0x221) [0x562ad6ffdf01]
8: (CInode::clear_client_caps_after_export()+0x3b) [0x562ad6ffe28b]
9: (Migrator::finish_export_inode_caps(CInode*, int, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >&)+0x534) [0x562ad6f86804]
10: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x282) [0x562ad6f87402]
11: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x41a) [0x562ad6f8759a]
12: (Migrator::export_finish(CDir*)+0x438) [0x562ad6f89988]
13: (Migrator::handle_export_notify_ack(boost::intrusive_ptr<MExportDirNotifyAck const> const&)+0x383) [0x562ad6f8abe3]
14: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x204) [0x562ad6f8b1f4]
15: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5b7) [0x562ad6d9f097]
16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x4f) [0x562ad6d9f68f]
17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x118) [0x562ad6d7b288]
18: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f7dd2842848]
19: (DispatchQueue::entry()+0x5ef) [0x7f7dd283ffef]
20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7dd28f7ecd]
21: (()+0x9432) [0x7f7dd1e68432]
22: (clone()+0x43) [0x7f7dd19c29d3]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-63> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command assert hook 0x562ad93c40f0
-62> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command abort hook 0x562ad93c40f0
-61> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command leak_some_memory hook 0x562ad93c40f0
-60> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perfcounters_dump hook 0x562ad93c40f0
-59> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command 1 hook 0x562ad93c40f0
-58> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perf dump hook 0x562ad93c40f0
-57> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perfcounters_schema hook 0x562ad93c40f0
-56> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perf histogram dump hook 0x562ad93c40f0
-55> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command 2 hook 0x562ad93c40f0
-54> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perf schema hook 0x562ad93c40f0
-53> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perf histogram schema hook 0x562ad93c40f0
-52> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command perf reset hook 0x562ad93c40f0
-51> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config show hook 0x562ad93c40f0
-50> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config help hook 0x562ad93c40f0
-49> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config set hook 0x562ad93c40f0
-48> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config unset hook 0x562ad93c40f0
-47> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config get hook 0x562ad93c40f0
-46> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config diff hook 0x562ad93c40f0
-45> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command config diff get hook 0x562ad93c40f0
-44> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command injectargs hook 0x562ad93c40f0
-43> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command log flush hook 0x562ad93c40f0
-42> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command log dump hook 0x562ad93c40f0
-41> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command log reopen hook 0x562ad93c40f0
-40> 2020-07-07T13:59:07.555+0000 7f7dd17f5600 5 asok(0x562ad945a000) register_command dump_mempools hook 0x562ada030068
-39> 2020-07-07T13:59:07.557+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
-38> 2020-07-07T13:59:07.559+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
-37> 2020-07-07T13:59:07.559+0000 7f7dd17f5600 0 ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev), process ceph-mds, pid 25530
-36> 2020-07-07T13:59:07.560+0000 7f7dd17f5600 0 pidfile_write: ignore empty --pid-file
-35> 2020-07-07T13:59:07.560+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
-34> 2020-07-07T13:59:07.569+0000 7f7dd17f5600 1 finished global_init_daemonize
-33> 2020-07-07T13:59:08.283+0000 7f7dccf8b700 4 mgrc handle_mgr_map Got map version 4
-32> 2020-07-07T13:59:08.283+0000 7f7dccf8b700 4 mgrc handle_mgr_map Active mgr is now [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
-31> 2020-07-07T13:59:08.283+0000 7f7dccf8b700 4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
-30> 2020-07-07T13:59:08.284+0000 7f7dccf8b700 4 mgrc handle_mgr_configure stats_period=5
-29> 2020-07-07T13:59:08.284+0000 7f7dccf8b700 4 mgrc handle_mgr_configure updated stats threshold: 5
-28> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x107
-27> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x646
-26> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x647
-25> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x648
-24> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x649
-23> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64a
-22> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64b
-21> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64c
-20> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64d
-19> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64e
-18> 2020-07-07T14:00:12.639+0000 7f7dccf8b700 0 mds.7.cache creating system inode with ino:0x64f
-17> 2020-07-07T14:29:08.284+0000 7f7dccf8b700 4 mgrc ms_handle_reset ms_handle_reset con 0x562ad94d9000
-16> 2020-07-07T14:29:08.284+0000 7f7dccf8b700 4 mgrc reconnect Terminating session with v2:10.0.1.1:6800/18339
-15> 2020-07-07T14:29:08.284+0000 7f7dccf8b700 4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
-14> 2020-07-07T14:29:08.284+0000 7f7dccf8b700 4 mgrc handle_mgr_configure stats_period=5
-13> 2020-07-07T14:44:08.284+0000 7f7dccf8b700 4 mgrc ms_handle_reset ms_handle_reset con 0x562ae12fd400
-12> 2020-07-07T14:44:08.284+0000 7f7dccf8b700 4 mgrc reconnect Terminating session with v2:10.0.1.1:6800/18339
-11> 2020-07-07T14:44:08.284+0000 7f7dccf8b700 4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
-10> 2020-07-07T14:44:08.284+0000 7f7dccf8b700 4 mgrc handle_mgr_configure stats_period=5
-9> 2020-07-07T14:59:08.284+0000 7f7dccf8b700 4 mgrc ms_handle_reset ms_handle_reset con 0x562ad94d9000
-8> 2020-07-07T14:59:08.284+0000 7f7dccf8b700 4 mgrc reconnect Terminating session with v2:10.0.1.1:6800/18339
-7> 2020-07-07T14:59:08.284+0000 7f7dccf8b700 4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
-6> 2020-07-07T14:59:08.284+0000 7f7dccf8b700 4 mgrc handle_mgr_configure stats_period=5
-5> 2020-07-07T15:14:08.285+0000 7f7dccf8b700 4 mgrc ms_handle_reset ms_handle_reset con 0x562ae08b6c00
-4> 2020-07-07T15:14:08.285+0000 7f7dccf8b700 4 mgrc reconnect Terminating session with v2:10.0.1.1:6800/18339
-3> 2020-07-07T15:14:08.285+0000 7f7dccf8b700 4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
-2> 2020-07-07T15:14:08.285+0000 7f7dccf8b700 4 mgrc handle_mgr_configure stats_period=5
-1> 2020-07-07T15:27:20.984+0000 7f7dccf8b700 -1 /home/fedora/src/ceph/ceph/src/include/elist.h: In function 'elist<T>::~elist() [with T = MDLockCache*]' thread 7f7dccf8b700 time 2020-07-07T15:27:20.979768+0000
/home/fedora/src/ceph/ceph/src/include/elist.h: 91: FAILED ceph_assert(_head.empty())
ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7f7dd262a774]
2: (()+0x2508ff) [0x7f7dd262a8ff]
3: (std::_Rb_tree<client_t, std::pair<client_t const, Capability>, std::_Select1st<std::pair<client_t const, Capability> >, std::less<client_t>, mempool::pool_allocator<(mempool::pool_index_t)18, std::pair<client_t const, Capability> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<client_t const, Capability> >)+0x15a) [0x562ad7020a4a]
4: (CInode::remove_client_cap(client_t)+0x221) [0x562ad6ffdf01]
5: (CInode::clear_client_caps_after_export()+0x3b) [0x562ad6ffe28b]
6: (Migrator::finish_export_inode_caps(CInode*, int, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >&)+0x534) [0x562ad6f86804]
7: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x282) [0x562ad6f87402]
8: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x41a) [0x562ad6f8759a]
9: (Migrator::export_finish(CDir*)+0x438) [0x562ad6f89988]
10: (Migrator::handle_export_notify_ack(boost::intrusive_ptr<MExportDirNotifyAck const> const&)+0x383) [0x562ad6f8abe3]
11: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x204) [0x562ad6f8b1f4]
12: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5b7) [0x562ad6d9f097]
13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x4f) [0x562ad6d9f68f]
14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x118) [0x562ad6d7b288]
15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f7dd2842848]
16: (DispatchQueue::entry()+0x5ef) [0x7f7dd283ffef]
17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7dd28f7ecd]
18: (()+0x9432) [0x7f7dd1e68432]
19: (clone()+0x43) [0x7f7dd19c29d3]
0> 2020-07-07T15:27:20.986+0000 7f7dccf8b700 -1 *** Caught signal (Aborted) **
in thread 7f7dccf8b700 thread_name:ms_dispatch
ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev)
1: (()+0x14a90) [0x7f7dd1e73a90]
2: (gsignal()+0x145) [0x7f7dd18fda25]
3: (abort()+0x127) [0x7f7dd18e6895]
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x7f7dd262a7be]
5: (()+0x2508ff) [0x7f7dd262a8ff]
6: (std::_Rb_tree<client_t, std::pair<client_t const, Capability>, std::_Select1st<std::pair<client_t const, Capability> >, std::less<client_t>, mempool::pool_allocator<(mempool::pool_index_t)18, std::pair<client_t const, Capability> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<client_t const, Capability> >)+0x15a) [0x562ad7020a4a]
7: (CInode::remove_client_cap(client_t)+0x221) [0x562ad6ffdf01]
8: (CInode::clear_client_caps_after_export()+0x3b) [0x562ad6ffe28b]
9: (Migrator::finish_export_inode_caps(CInode*, int, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >&)+0x534) [0x562ad6f86804]
10: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x282) [0x562ad6f87402]
11: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x41a) [0x562ad6f8759a]
12: (Migrator::export_finish(CDir*)+0x438) [0x562ad6f89988]
13: (Migrator::handle_export_notify_ack(boost::intrusive_ptr<MExportDirNotifyAck const> const&)+0x383) [0x562ad6f8abe3]
14: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x204) [0x562ad6f8b1f4]
15: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5b7) [0x562ad6d9f097]
16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x4f) [0x562ad6d9f68f]
17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x118) [0x562ad6d7b288]
18: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f7dd2842848]
19: (DispatchQueue::entry()+0x5ef) [0x7f7dd283ffef]
20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7dd28f7ecd]
21: (()+0x9432) [0x7f7dd1e68432]
22: (clone()+0x43) [0x7f7dd19c29d3]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 0 lockdep
0/ 0 context
0/ 0 crush
0/ 0 mds
0/ 0 mds_balancer
0/ 0 mds_locker
0/ 0 mds_log
0/ 0 mds_log_expire
0/ 0 mds_migrator
0/ 0 buffer
0/ 0 timer
0/ 0 filer
0/ 1 striper
0/ 0 objecter
0/ 0 rados
0/ 0 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_rwl
0/ 0 journaler
0/ 0 objectcacher
0/ 5 immutable_obj_cache
0/ 0 client
0/ 0 osd
0/ 0 optracker
0/ 0 objclass
0/ 0 filestore
0/ 0 journal
0/ 0 ms
0/ 0 mon
0/ 0 monc
0/ 0 paxos
0/ 0 tp
0/ 0 auth
1/ 5 crypto
0/ 0 finisher
1/ 1 reserver
0/ 0 heartbeatmap
0/ 0 perfcounter
0/ 0 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
0/ 0 asok
0/ 0 throttle
0/ 0 refs
1/ 5 compressor
5/ 5 bluestore
0/ 0 bluefs
0/ 0 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
5/ 5 prioritycache
0/ 5 test
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
7f7dccf8b700 / ms_dispatch
7f7dd17f5600 / ceph-mds
max_recent 10000
max_new 1000
log_file /tmp/cbt/ceph/log/mds.d.log
--- end dump of recent events ---
</pre>
CephFS - Bug #44294: mds: "elist.h: 91: FAILED ceph_assert(_head.empty())"
https://tracker.ceph.com/issues/44294?journal_id=172006
2020-07-30T17:43:43Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Pending Backport</i></li><li><strong>Backport</strong> set to <i>nautilus</i></li></ul><p>This should have been flagged for backport.</p>
CephFS - Bug #44294: mds: "elist.h: 91: FAILED ceph_assert(_head.empty())"
https://tracker.ceph.com/issues/44294?journal_id=172007
2020-07-30T17:44:34Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-10 priority-6 priority-high2 closed" href="/issues/46778">Backport #46778</a>: nautilus: mds: "elist.h: 91: FAILED ceph_assert(_head.empty())"</i> added</li></ul>
CephFS - Bug #44294: mds: "elist.h: 91: FAILED ceph_assert(_head.empty())"
https://tracker.ceph.com/issues/44294?journal_id=172015
2020-07-30T18:52:06Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>
CephFS - Bug #44294: mds: "elist.h: 91: FAILED ceph_assert(_head.empty())"
https://tracker.ceph.com/issues/44294?journal_id=172922
2020-08-14T18:45:50Z
Nathan Cutler
ncutler@suse.cz
<ul></ul><p>This issue and <a class="issue tracker-1 status-3 priority-6 priority-high2 closed" title="Bug: mds: MDCache.cc: 6400: FAILED ceph_assert(r == 0 || r == -2) (Resolved)" href="https://tracker.ceph.com/issues/44295">#44295</a> were both fixed by the same PR, <a class="external" href="https://github.com/ceph/ceph/pull/33538">https://github.com/ceph/ceph/pull/33538</a>, which was backported to nautilus via <a class="issue tracker-1 status-3 priority-6 priority-high2 closed" title="Bug: mds: MDCache.cc: 6400: FAILED ceph_assert(r == 0 || r == -2) (Resolved)" href="https://tracker.ceph.com/issues/44295">#44295</a>.</p>