Bug #53750
mds: FAILED ceph_assert(mut->is_wrlocked(&pin->filelock))
% Done:
0%
Source:
Tags:
Backport:
pacific,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
fs
Component(FS):
MDS
Labels (FS):
crash
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
10101 -7> 2021-12-31T21:08:02.851+0800 1461ab3b1700 4 mds.0.server handle_client_request client_request(client.4217:33 lookup #0x1/mydir 2021-12-3 1T21:08:02.851772+0800 caller_uid=0, caller_gid=0{0,}) v5 10102 -6> 2021-12-31T21:08:02.853+0800 1461ab3b1700 4 mds.0.server handle_client_request client_request(client.4242:5209 unlink #0x10000000000/fil e0 2021-12-31T21:08:02.853439+0800 caller_uid=0, caller_gid=0{0,}) v5 10103 -5> 2021-12-31T21:08:02.857+0800 1461ab3b1700 4 mds.0.server handle_client_request client_request(client.4217:34 lookup #0x10000000000/file1 2021-12-31T21:08:02.857737+0800 caller_uid=0, caller_gid=0{0,}) v5 10104 -4> 2021-12-31T21:08:02.859+0800 1461a99a4700 5 mds.0.log _submit_thread 28124832~3114 : EUpdate unlink_local [metablob 0x1, 4 dirs] 10105 -3> 2021-12-31T21:08:02.859+0800 1461a99a4700 5 mds.0.log _submit_thread 28127966~872 : ESubtreeMap 2 subtrees , 0 ambiguous [metablob 0x1, 2 dirs] 10106 -2> 2021-12-31T21:08:02.861+0800 1461ab3b1700 4 mds.0.server handle_client_request client_request(client.4217:35 link #0x10000000000/file1 # 0x10000000cd7 2021-12-31T21:08:02.861659+0800 caller_uid=0, caller_gid=0{0,}) v5 10107 -1> 2021-12-31T21:08:02.889+0800 1461a9ba5700 -1 /data/ceph/src/mds/MDCache.cc: In function 'void MDCache::predirty_journal_parents(MutationR ef, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)' thread 1461a9ba5700 time 2021-12-31T21:08:02.881576+0800 10108 /data/ceph/src/mds/MDCache.cc: 2168: FAILED ceph_assert(mut->is_wrlocked(&pin->filelock)) 10109 10110 ceph version 17.0.0-9829-g20d7b3fb38d (20d7b3fb38dec666cd6076841f72374f7aa945d2) quincy (dev) 10111 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x1461b099b929] 10112 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x1461b099bb5b] 10113 3: (MDCache::predirty_journal_parents(boost::intrusive_ptr<MutationImpl>, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0x5db) [0x55832711649d ] 10114 4: (Server::_link_local(boost::intrusive_ptr<MDRequestImpl>&, CDentry*, CInode*, SnapRealm*)+0x5b6) [0x55832707cd9e] 10115 5: (Server::handle_client_link(boost::intrusive_ptr<MDRequestImpl>&)+0x1356) [0x55832707fc68] 10116 6: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x13e9) [0x5583270afa97] 10117 7: (MDCache::dispatch_request(boost::intrusive_ptr<MDRequestImpl>&)+0x33) [0x55832718d4b3] 10118 8: (C_MDS_RetryRequest::finish(int)+0x2c) [0x55832718d5fa] 10119 9: (MDSContext::complete(int)+0x7e) [0x55832734ac22] 10120 10: (MDSCacheObject::finish_waiting(unsigned long, int)+0xd7a) [0x5583273780ae] 10121 11: (SimpleLock::finish_waiters(unsigned long, int)+0x26) [0x55832722100e] 10122 12: (Locker::eval_gather(SimpleLock*, bool, bool*, std::vector<MDSContext*, std::allocator<MDSContext*> >*)+0x16ff) [0x55832720a04f] 10123 13: (Locker::_finish_xlock(SimpleLock*, client_t, bool*)+0xbd) [0x55832720a30f] 10124 14: (Locker::xlock_finish(std::_Rb_tree_const_iterator<MutationImpl::LockOp> const&, MutationImpl*, bool*)+0x4fa) [0x558327210bb4] 10125 15: (Locker::_drop_locks(MutationImpl*, std::set<CInode*, std::less<CInode*>, std::allocator<CInode*> >*, bool)+0x96) [0x558327210c92] 10126 16: (Locker::drop_non_rdlocks(MutationImpl*, std::set<CInode*, std::less<CInode*>, std::allocator<CInode*> >*)+0x5a) [0x5583272125a8] 10127 17: (MDCache::request_drop_non_rdlocks(boost::intrusive_ptr<MDRequestImpl>&)+0x26) [0x5583271325da] 10128 18: (Server::reply_client_request(boost::intrusive_ptr<MDRequestImpl>&, boost::intrusive_ptr<MClientReply> const&)+0x3a1) [0x55832705dae3] 10129 19: (Server::respond_to_request(boost::intrusive_ptr<MDRequestImpl>&, int)+0x295) [0x55832705eb9d] 10130 20: (Server::_unlink_local_finish(boost::intrusive_ptr<MDRequestImpl>&, CDentry*, CDentry*, unsigned long)+0x597) [0x558327084c8b] 10131 21: (C_MDS_unlink_local_finish::finish(int)+0x24) [0x5583270fad4c] 10132 22: (MDSContext::complete(int)+0x7e) [0x55832734ac22] 10133 23: (MDSIOContextBase::complete(int)+0x19fa) [0x55832734ccd4] 10134 24: (MDSLogContextBase::complete(int)+0x47) [0x55832734d02b] 10135 25: (Finisher::finisher_thread_entry()+0x116b) [0x1461b0916225] 10136 26: (Finisher::FinisherThread::entry()+0xd) [0x558327022a55] 10137 27: (Thread::entry_wrapper()+0x3f) [0x1461b096e497] 10138 28: (Thread::_entry_func(void*)+0x9) [0x1461b096e4af] 10139 29: /lib64/libpthread.so.0(+0x82de) [0x1461af5a82de] 10140 30: clone() 10141 10142 0> 2021-12-31T21:08:02.899+0800 1461a9ba5700 -1 *** Caught signal (Aborted) ** 10143 in thread 1461a9ba5700 thread_name:MR_Finisher 10144
Related issues
History
#1 Updated by Xiubo Li about 2 years ago
- Status changed from New to In Progress
#2 Updated by Xiubo Li about 2 years ago
This is reproducable very easily by using the following scripts in two different teriminal:
$ while [ 1 ]; do rm -rf /mnt/cephfs/mydir/file1; ln /mnt/cephfs/mydir/file0 /mnt/cephfs/mydir/file1; echo '---------'; done $ while [ 1 ]; do touch /mnt/kcephfs/mydir/file0; rm -rf /mnt/kcephfs/mydir/file0; echo "----"; done
#3 Updated by Xiubo Li about 2 years ago
- Priority changed from Normal to High
#4 Updated by Xiubo Li about 2 years ago
- Status changed from In Progress to Fix Under Review
- Backport changed from pacific to pacific,octopus
- Pull request ID set to 44451
#5 Updated by Venky Shankar about 2 years ago
- Status changed from Fix Under Review to Pending Backport
#6 Updated by Backport Bot about 2 years ago
- Copied to Backport #53864: pacific: mds: FAILED ceph_assert(mut->is_wrlocked(&pin->filelock)) added
#7 Updated by Backport Bot about 2 years ago
- Copied to Backport #53865: octopus: mds: FAILED ceph_assert(mut->is_wrlocked(&pin->filelock)) added
#8 Updated by Xiubo Li about 2 years ago
- Status changed from Pending Backport to Resolved