Bug #18755: multimds: MDCache.cc: 4735: FAILED assert(in) - CephFS - Ceph

Actions

#1

Updated by Patrick Donnelly over 7 years ago

There are actually 3 assertion failures (on 3 different MDS) in this run!

2017-01-31T17:24:17.908 INFO:tasks.ceph.mds.a.mira037.stderr:/mnt/jenkins/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/11.1.0-7091-g326a113/rpm/el7/BUILD/ceph-11.1.0-7091-g326a113/src/mds/MDCache.cc: 328: FAILED assert(o->get_num_ref() == 0)
2017-01-31T17:24:17.909 INFO:tasks.ceph.mds.a.mira037.stderr: ceph version 11.1.0-7091-g326a113 (326a11362f76098867add4557a03a45a219a640c)
2017-01-31T17:24:17.909 INFO:tasks.ceph.mds.a.mira037.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55b317365690]
2017-01-31T17:24:17.910 INFO:tasks.ceph.mds.a.mira037.stderr: 2: (MDCache::remove_inode(CInode*)+0x43d) [0x55b31711cd2d]
2017-01-31T17:24:17.910 INFO:tasks.ceph.mds.a.mira037.stderr: 3: (StrayManager::_purge_stray_logged(CDentry*, unsigned long, LogSegment*)+0x208) [0x55b3171b45f8]
2017-01-31T17:24:17.911 INFO:tasks.ceph.mds.a.mira037.stderr: 4: (MDSIOContextBase::complete(int)+0xa4) [0x55b3172b31f4]
2017-01-31T17:24:17.911 INFO:tasks.ceph.mds.a.mira037.stderr: 5: (MDSLogContextBase::complete(int)+0x3c) [0x55b3172b362c]
2017-01-31T17:24:17.911 INFO:tasks.ceph.mds.a.mira037.stderr: 6: (Finisher::finisher_thread_entry()+0x1ee) [0x55b3173622be]
2017-01-31T17:24:17.911 INFO:tasks.ceph.mds.a.mira037.stderr: 7: (()+0x7dc5) [0x7f306ebebdc5]
2017-01-31T17:24:17.912 INFO:tasks.ceph.mds.a.mira037.stderr: 8: (clone()+0x6d) [0x7f306dee373d]

2017-01-31T17:24:37.610 INFO:tasks.ceph.mds.e-s.mira035.stderr:/mnt/jenkins/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/11.1.0-7091-g326a113/rpm/el7/BUILD/ceph-11.1.0-7091-g326a113/src/mds/MDCache.cc: 4735: FAILED assert(in)
2017-01-31T17:24:37.614 INFO:tasks.ceph.mds.e-s.mira035.stderr: ceph version 11.1.0-7091-g326a113 (326a11362f76098867add4557a03a45a219a640c)
2017-01-31T17:24:37.614 INFO:tasks.ceph.mds.e-s.mira035.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55863fa32690]
2017-01-31T17:24:37.614 INFO:tasks.ceph.mds.e-s.mira035.stderr: 2: (MDCache::handle_cache_rejoin_strong(MMDSCacheRejoin*)+0x20e0) [0x55863f836af0]
2017-01-31T17:24:37.614 INFO:tasks.ceph.mds.e-s.mira035.stderr: 3: (MDCache::handle_cache_rejoin(MMDSCacheRejoin*)+0x203) [0x55863f837f63]
2017-01-31T17:24:37.614 INFO:tasks.ceph.mds.e-s.mira035.stderr: 4: (MDCache::dispatch(Message*)+0xa5) [0x55863f83d385]
2017-01-31T17:24:37.614 INFO:tasks.ceph.mds.e-s.mira035.stderr: 5: (MDSRank::handle_deferrable_message(Message*)+0x5bc) [0x55863f7335cc]
2017-01-31T17:24:37.615 INFO:tasks.ceph.mds.e-s.mira035.stderr: 6: (MDSRank::_dispatch(Message*, bool)+0x20c) [0x55863f73cdcc]
2017-01-31T17:24:37.615 INFO:tasks.ceph.mds.e-s.mira035.stderr: 7: (MDSRankDispatcher::ms_dispatch(Message*)+0x15) [0x55863f73df95]
2017-01-31T17:24:37.615 INFO:tasks.ceph.mds.e-s.mira035.stderr: 8: (MDSDaemon::ms_dispatch(Message*)+0xf3) [0x55863f72b803]
2017-01-31T17:24:37.615 INFO:tasks.ceph.mds.e-s.mira035.stderr: 9: (DispatchQueue::entry()+0x7a2) [0x55863fc409d2]
2017-01-31T17:24:37.615 INFO:tasks.ceph.mds.e-s.mira035.stderr: 10: (DispatchQueue::DispatchThread::entry()+0xd) [0x55863faa275d]
2017-01-31T17:24:37.615 INFO:tasks.ceph.mds.e-s.mira035.stderr: 11: (()+0x7dc5) [0x7fed38e7edc5]
2017-01-31T17:24:37.615 INFO:tasks.ceph.mds.e-s.mira035.stderr: 12: (clone()+0x6d) [0x7fed3817673d]

2017-01-31T17:25:06.448 INFO:tasks.ceph.mds.c.mira035.stderr:/mnt/jenkins/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/11.1.0-7091-g326a113/rpm/el7/BUILD/ceph-11.1.0-7091-g326a113/src/mds/MDCache.cc: 4735: FAILED assert(in)
2017-01-31T17:25:06.453 INFO:tasks.ceph.mds.c.mira035.stderr: ceph version 11.1.0-7091-g326a113 (326a11362f76098867add4557a03a45a219a640c)
2017-01-31T17:25:06.453 INFO:tasks.ceph.mds.c.mira035.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x5639ff555690]
2017-01-31T17:25:06.453 INFO:tasks.ceph.mds.c.mira035.stderr: 2: (MDCache::handle_cache_rejoin_strong(MMDSCacheRejoin*)+0x20e0) [0x5639ff359af0]
2017-01-31T17:25:06.453 INFO:tasks.ceph.mds.c.mira035.stderr: 3: (MDCache::handle_cache_rejoin(MMDSCacheRejoin*)+0x203) [0x5639ff35af63]
2017-01-31T17:25:06.453 INFO:tasks.ceph.mds.c.mira035.stderr: 4: (MDCache::dispatch(Message*)+0xa5) [0x5639ff360385]
2017-01-31T17:25:06.453 INFO:tasks.ceph.mds.c.mira035.stderr: 5: (MDSRank::handle_deferrable_message(Message*)+0x5bc) [0x5639ff2565cc]
2017-01-31T17:25:06.453 INFO:tasks.ceph.mds.c.mira035.stderr: 6: (MDSRank::_dispatch(Message*, bool)+0x20c) [0x5639ff25fdcc]
2017-01-31T17:25:06.454 INFO:tasks.ceph.mds.c.mira035.stderr: 7: (MDSRankDispatcher::ms_dispatch(Message*)+0x15) [0x5639ff260f95]
2017-01-31T17:25:06.454 INFO:tasks.ceph.mds.c.mira035.stderr: 8: (MDSDaemon::ms_dispatch(Message*)+0xf3) [0x5639ff24e803]
2017-01-31T17:25:06.454 INFO:tasks.ceph.mds.c.mira035.stderr: 9: (DispatchQueue::entry()+0x7a2) [0x5639ff7639d2]
2017-01-31T17:25:06.454 INFO:tasks.ceph.mds.c.mira035.stderr: 10: (DispatchQueue::DispatchThread::entry()+0xd) [0x5639ff5c575d]
2017-01-31T17:25:06.454 INFO:tasks.ceph.mds.c.mira035.stderr: 11: (()+0x7dc5) [0x7f62f41d6dc5]
2017-01-31T17:25:06.454 INFO:tasks.ceph.mds.c.mira035.stderr: 12: (clone()+0x6d) [0x7f62f34ce73d]

Actions

Copy link

#2

Updated by Zheng Yan over 7 years ago

 -1189> 2017-01-31 17:24:17.897818 7f306a254700  7 mds.2.cache handle_discover added dentry [dentry #102/stray1/2000000002c [2,head] auth{0=1} (dversion lock) pv=1277 v=353 inode=0x55b320eb5d88 | request=0 lock=0 inodepin=0 purging=1 replicated=1 dirty=0 authpin=0 0x55b320f15450]

MDCache::handle_discover adds replica to purging inode. This can explain all assertions

Actions

Copy link

#3