Project

General

Profile

Actions
Copy link

Bug #46403

open

mds: "elist.h: 91: FAILED ceph_assert(_head.empty())"

Added by Mark Nelson almost 4 years ago. Updated almost 2 years ago.

Status:
Triaged
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
pacific,octopus,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This may be a re-occurance of https://tracker.ceph.com/issues/44294

Saw this while testing ceph.dir.pin.distributed for the IO500 challenge.

2020-07-07T13:59:07.557+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
2020-07-07T13:59:07.559+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
2020-07-07T13:59:07.559+0000 7f7dd17f5600  0 ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev), process ceph-mds, pid 25530
2020-07-07T13:59:07.560+0000 7f7dd17f5600  0 pidfile_write: ignore empty --pid-file
2020-07-07T13:59:07.560+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x107
2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x646
2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x647
2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x648
2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x649
2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x64a
2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x64b
2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x64c
2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x64d
2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x64e
2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x64f
2020-07-07T15:27:20.984+0000 7f7dccf8b700 -1 /home/fedora/src/ceph/ceph/src/include/elist.h: In function 'elist<T>::~elist() [with T = MDLockCache*]' thread 7f7dccf8b700 time 2020-07-07T15:27:20.979768+0000
/home/fedora/src/ceph/ceph/src/include/elist.h: 91: FAILED ceph_assert(_head.empty())

 ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7f7dd262a774]
 2: (()+0x2508ff) [0x7f7dd262a8ff]
 3: (std::_Rb_tree<client_t, std::pair<client_t const, Capability>, std::_Select1st<std::pair<client_t const, Capability> >, std::less<client_t>, mempool::pool_allocator<(mempool::pool_index_t)18, std::pair<client_t const, Capability> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<client_t const, Capability> >)+0x15a) [0x562ad7020a4a]
 4: (CInode::remove_client_cap(client_t)+0x221) [0x562ad6ffdf01]
 5: (CInode::clear_client_caps_after_export()+0x3b) [0x562ad6ffe28b]
 6: (Migrator::finish_export_inode_caps(CInode*, int, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >&)+0x534) [0x562ad6f86804]
 7: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x282) [0x562ad6f87402]
 8: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x41a) [0x562ad6f8759a]
 9: (Migrator::export_finish(CDir*)+0x438) [0x562ad6f89988]
 10: (Migrator::handle_export_notify_ack(boost::intrusive_ptr<MExportDirNotifyAck const> const&)+0x383) [0x562ad6f8abe3]
 11: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x204) [0x562ad6f8b1f4]
 12: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5b7) [0x562ad6d9f097]
 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x4f) [0x562ad6d9f68f]
 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x118) [0x562ad6d7b288]
 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f7dd2842848]
 16: (DispatchQueue::entry()+0x5ef) [0x7f7dd283ffef]
 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7dd28f7ecd]
 18: (()+0x9432) [0x7f7dd1e68432]
 19: (clone()+0x43) [0x7f7dd19c29d3]

2020-07-07T15:27:20.986+0000 7f7dccf8b700 -1 *** Caught signal (Aborted) **
 in thread 7f7dccf8b700 thread_name:ms_dispatch

 ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev)
 1: (()+0x14a90) [0x7f7dd1e73a90]
 2: (gsignal()+0x145) [0x7f7dd18fda25]
 3: (abort()+0x127) [0x7f7dd18e6895]
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x7f7dd262a7be]
 5: (()+0x2508ff) [0x7f7dd262a8ff]
 6: (std::_Rb_tree<client_t, std::pair<client_t const, Capability>, std::_Select1st<std::pair<client_t const, Capability> >, std::less<client_t>, mempool::pool_allocator<(mempool::pool_index_t)18, std::pair<client_t const, Capability> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<client_t const, Capability> >)+0x15a) [0x562ad7020a4a]
 7: (CInode::remove_client_cap(client_t)+0x221) [0x562ad6ffdf01]
 8: (CInode::clear_client_caps_after_export()+0x3b) [0x562ad6ffe28b]
 9: (Migrator::finish_export_inode_caps(CInode*, int, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >&)+0x534) [0x562ad6f86804]
 10: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x282) [0x562ad6f87402]
 11: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x41a) [0x562ad6f8759a]
 12: (Migrator::export_finish(CDir*)+0x438) [0x562ad6f89988]
 13: (Migrator::handle_export_notify_ack(boost::intrusive_ptr<MExportDirNotifyAck const> const&)+0x383) [0x562ad6f8abe3]
 14: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x204) [0x562ad6f8b1f4]
 15: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5b7) [0x562ad6d9f097]
 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x4f) [0x562ad6d9f68f]
 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x118) [0x562ad6d7b288]
 18: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f7dd2842848]
 19: (DispatchQueue::entry()+0x5ef) [0x7f7dd283ffef]
 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7dd28f7ecd]
 21: (()+0x9432) [0x7f7dd1e68432]
 22: (clone()+0x43) [0x7f7dd19c29d3]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -63> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command assert hook 0x562ad93c40f0
   -62> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command abort hook 0x562ad93c40f0
   -61> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command leak_some_memory hook 0x562ad93c40f0
   -60> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command perfcounters_dump hook 0x562ad93c40f0
   -59> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command 1 hook 0x562ad93c40f0
   -58> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command perf dump hook 0x562ad93c40f0
   -57> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command perfcounters_schema hook 0x562ad93c40f0
   -56> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command perf histogram dump hook 0x562ad93c40f0
   -55> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command 2 hook 0x562ad93c40f0
   -54> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command perf schema hook 0x562ad93c40f0
   -53> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command perf histogram schema hook 0x562ad93c40f0
   -52> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command perf reset hook 0x562ad93c40f0
   -51> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command config show hook 0x562ad93c40f0
   -50> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command config help hook 0x562ad93c40f0
   -49> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command config set hook 0x562ad93c40f0
   -48> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command config unset hook 0x562ad93c40f0
   -47> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command config get hook 0x562ad93c40f0
   -46> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command config diff hook 0x562ad93c40f0
   -45> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command config diff get hook 0x562ad93c40f0
   -44> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command injectargs hook 0x562ad93c40f0
   -43> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command log flush hook 0x562ad93c40f0
   -42> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command log dump hook 0x562ad93c40f0
   -41> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command log reopen hook 0x562ad93c40f0
   -40> 2020-07-07T13:59:07.555+0000 7f7dd17f5600  5 asok(0x562ad945a000) register_command dump_mempools hook 0x562ada030068
   -39> 2020-07-07T13:59:07.557+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
   -38> 2020-07-07T13:59:07.559+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
   -37> 2020-07-07T13:59:07.559+0000 7f7dd17f5600  0 ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev), process ceph-mds, pid 25530
   -36> 2020-07-07T13:59:07.560+0000 7f7dd17f5600  0 pidfile_write: ignore empty --pid-file
   -35> 2020-07-07T13:59:07.560+0000 7f7dd17f5600 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,lmdb,rocksdb
   -34> 2020-07-07T13:59:07.569+0000 7f7dd17f5600  1 finished global_init_daemonize
   -33> 2020-07-07T13:59:08.283+0000 7f7dccf8b700  4 mgrc handle_mgr_map Got map version 4
   -32> 2020-07-07T13:59:08.283+0000 7f7dccf8b700  4 mgrc handle_mgr_map Active mgr is now [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
   -31> 2020-07-07T13:59:08.283+0000 7f7dccf8b700  4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
   -30> 2020-07-07T13:59:08.284+0000 7f7dccf8b700  4 mgrc handle_mgr_configure stats_period=5
   -29> 2020-07-07T13:59:08.284+0000 7f7dccf8b700  4 mgrc handle_mgr_configure updated stats threshold: 5
   -28> 2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x107
   -27> 2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x646
   -26> 2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x647
   -25> 2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x648
   -24> 2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x649
   -23> 2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x64a
   -22> 2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x64b
   -21> 2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x64c
   -20> 2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x64d
   -19> 2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x64e
   -18> 2020-07-07T14:00:12.639+0000 7f7dccf8b700  0 mds.7.cache creating system inode with ino:0x64f
   -17> 2020-07-07T14:29:08.284+0000 7f7dccf8b700  4 mgrc ms_handle_reset ms_handle_reset con 0x562ad94d9000
   -16> 2020-07-07T14:29:08.284+0000 7f7dccf8b700  4 mgrc reconnect Terminating session with v2:10.0.1.1:6800/18339
   -15> 2020-07-07T14:29:08.284+0000 7f7dccf8b700  4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
   -14> 2020-07-07T14:29:08.284+0000 7f7dccf8b700  4 mgrc handle_mgr_configure stats_period=5
   -13> 2020-07-07T14:44:08.284+0000 7f7dccf8b700  4 mgrc ms_handle_reset ms_handle_reset con 0x562ae12fd400
   -12> 2020-07-07T14:44:08.284+0000 7f7dccf8b700  4 mgrc reconnect Terminating session with v2:10.0.1.1:6800/18339
   -11> 2020-07-07T14:44:08.284+0000 7f7dccf8b700  4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
   -10> 2020-07-07T14:44:08.284+0000 7f7dccf8b700  4 mgrc handle_mgr_configure stats_period=5
    -9> 2020-07-07T14:59:08.284+0000 7f7dccf8b700  4 mgrc ms_handle_reset ms_handle_reset con 0x562ad94d9000
    -8> 2020-07-07T14:59:08.284+0000 7f7dccf8b700  4 mgrc reconnect Terminating session with v2:10.0.1.1:6800/18339
    -7> 2020-07-07T14:59:08.284+0000 7f7dccf8b700  4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
    -6> 2020-07-07T14:59:08.284+0000 7f7dccf8b700  4 mgrc handle_mgr_configure stats_period=5
    -5> 2020-07-07T15:14:08.285+0000 7f7dccf8b700  4 mgrc ms_handle_reset ms_handle_reset con 0x562ae08b6c00
    -4> 2020-07-07T15:14:08.285+0000 7f7dccf8b700  4 mgrc reconnect Terminating session with v2:10.0.1.1:6800/18339
    -3> 2020-07-07T15:14:08.285+0000 7f7dccf8b700  4 mgrc reconnect Starting new session with [v2:10.0.1.1:6800/18339,v1:10.0.1.1:6801/18339]
    -2> 2020-07-07T15:14:08.285+0000 7f7dccf8b700  4 mgrc handle_mgr_configure stats_period=5
    -1> 2020-07-07T15:27:20.984+0000 7f7dccf8b700 -1 /home/fedora/src/ceph/ceph/src/include/elist.h: In function 'elist<T>::~elist() [with T = MDLockCache*]' thread 7f7dccf8b700 time 2020-07-07T15:27:20.979768+0000
/home/fedora/src/ceph/ceph/src/include/elist.h: 91: FAILED ceph_assert(_head.empty())

 ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7f7dd262a774]
 2: (()+0x2508ff) [0x7f7dd262a8ff]
 3: (std::_Rb_tree<client_t, std::pair<client_t const, Capability>, std::_Select1st<std::pair<client_t const, Capability> >, std::less<client_t>, mempool::pool_allocator<(mempool::pool_index_t)18, std::pair<client_t const, Capability> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<client_t const, Capability> >)+0x15a) [0x562ad7020a4a]
 4: (CInode::remove_client_cap(client_t)+0x221) [0x562ad6ffdf01]
 5: (CInode::clear_client_caps_after_export()+0x3b) [0x562ad6ffe28b]
 6: (Migrator::finish_export_inode_caps(CInode*, int, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >&)+0x534) [0x562ad6f86804]
 7: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x282) [0x562ad6f87402]
 8: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x41a) [0x562ad6f8759a]
 9: (Migrator::export_finish(CDir*)+0x438) [0x562ad6f89988]
 10: (Migrator::handle_export_notify_ack(boost::intrusive_ptr<MExportDirNotifyAck const> const&)+0x383) [0x562ad6f8abe3]
 11: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x204) [0x562ad6f8b1f4]
 12: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5b7) [0x562ad6d9f097]
 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x4f) [0x562ad6d9f68f]
 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x118) [0x562ad6d7b288]
 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f7dd2842848]
 16: (DispatchQueue::entry()+0x5ef) [0x7f7dd283ffef]
 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7dd28f7ecd]
 18: (()+0x9432) [0x7f7dd1e68432]
 19: (clone()+0x43) [0x7f7dd19c29d3]

     0> 2020-07-07T15:27:20.986+0000 7f7dccf8b700 -1 *** Caught signal (Aborted) **
 in thread 7f7dccf8b700 thread_name:ms_dispatch

 ceph version 16.0.0-3204-ge9cc7d863a (e9cc7d863ac8d63caefd191d9b7942e51c5bf780) pacific (dev)
 1: (()+0x14a90) [0x7f7dd1e73a90]
 2: (gsignal()+0x145) [0x7f7dd18fda25]
 3: (abort()+0x127) [0x7f7dd18e6895]
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x7f7dd262a7be]
 5: (()+0x2508ff) [0x7f7dd262a8ff]
 6: (std::_Rb_tree<client_t, std::pair<client_t const, Capability>, std::_Select1st<std::pair<client_t const, Capability> >, std::less<client_t>, mempool::pool_allocator<(mempool::pool_index_t)18, std::pair<client_t const, Capability> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<client_t const, Capability> >)+0x15a) [0x562ad7020a4a]
 7: (CInode::remove_client_cap(client_t)+0x221) [0x562ad6ffdf01]
 8: (CInode::clear_client_caps_after_export()+0x3b) [0x562ad6ffe28b]
 9: (Migrator::finish_export_inode_caps(CInode*, int, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >&)+0x534) [0x562ad6f86804]
 10: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x282) [0x562ad6f87402]
 11: (Migrator::finish_export_dir(CDir*, int, std::map<inodeno_t, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > >, std::less<inodeno_t>, std::allocator<std::pair<inodeno_t const, std::map<client_t, Capability::Import, std::less<client_t>, std::allocator<std::pair<client_t const, Capability::Import> > > > > >&, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int*)+0x41a) [0x562ad6f8759a]
 12: (Migrator::export_finish(CDir*)+0x438) [0x562ad6f89988]
 13: (Migrator::handle_export_notify_ack(boost::intrusive_ptr<MExportDirNotifyAck const> const&)+0x383) [0x562ad6f8abe3]
 14: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x204) [0x562ad6f8b1f4]
 15: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5b7) [0x562ad6d9f097]
 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x4f) [0x562ad6d9f68f]
 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x118) [0x562ad6d7b288]
 18: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f7dd2842848]
 19: (DispatchQueue::entry()+0x5ef) [0x7f7dd283ffef]
 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f7dd28f7ecd]
 21: (()+0x9432) [0x7f7dd1e68432]
 22: (clone()+0x43) [0x7f7dd19c29d3]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   0/ 0 mds_balancer
   0/ 0 mds_locker
   0/ 0 mds_log
   0/ 0 mds_log_expire
   0/ 0 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 rbd_rwl
   0/ 0 journaler
   0/ 0 objectcacher
   0/ 5 immutable_obj_cache
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   1/ 1 reserver
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/ 5 rgw_sync
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 compressor
   5/ 5 bluestore
   0/ 0 bluefs
   0/ 0 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
   5/ 5 prioritycache
   0/ 5 test
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
  7f7dccf8b700 / ms_dispatch
  7f7dd17f5600 / ceph-mds
  max_recent     10000
  max_new         1000
  log_file /tmp/cbt/ceph/log/mds.d.log
--- end dump of recent events ---
Actions
Copy link
 #1

Updated by Patrick Donnelly almost 4 years ago

  • Status changed from New to Triaged
  • Assignee set to Zheng Yan
  • Target version set to v16.0.0
  • Source set to Development
  • Backport set to octopus
  • Component(FS) MDS added

Not related to #44294.

Zheng, this looks like a lock cache bug.

Actions
Copy link
 #2

Updated by Patrick Donnelly over 3 years ago

  • Assignee deleted (Zheng Yan)
Actions
Copy link
 #3

Updated by Patrick Donnelly over 3 years ago

  • Target version changed from v16.0.0 to v17.0.0
  • Backport changed from octopus to pacific,octopus,nautilus
Actions
Copy link
 #4

Updated by Patrick Donnelly almost 2 years ago

  • Target version deleted (v17.0.0)
Actions
Copy link

Also available in: Atom PDF