Project

General

Profile

Bug #53750

mds: FAILED ceph_assert(mut->is_wrlocked(&pin->filelock))

Added by Xiubo Li about 2 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
pacific,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
fs
Component(FS):
MDS
Labels (FS):
crash
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

10101     -7> 2021-12-31T21:08:02.851+0800 1461ab3b1700  4 mds.0.server handle_client_request client_request(client.4217:33 lookup #0x1/mydir 2021-12-3      1T21:08:02.851772+0800 caller_uid=0, caller_gid=0{0,}) v5
10102     -6> 2021-12-31T21:08:02.853+0800 1461ab3b1700  4 mds.0.server handle_client_request client_request(client.4242:5209 unlink #0x10000000000/fil      e0 2021-12-31T21:08:02.853439+0800 caller_uid=0, caller_gid=0{0,}) v5
10103     -5> 2021-12-31T21:08:02.857+0800 1461ab3b1700  4 mds.0.server handle_client_request client_request(client.4217:34 lookup #0x10000000000/file1       2021-12-31T21:08:02.857737+0800 caller_uid=0, caller_gid=0{0,}) v5
10104     -4> 2021-12-31T21:08:02.859+0800 1461a99a4700  5 mds.0.log _submit_thread 28124832~3114 : EUpdate unlink_local [metablob 0x1, 4 dirs]
10105     -3> 2021-12-31T21:08:02.859+0800 1461a99a4700  5 mds.0.log _submit_thread 28127966~872 : ESubtreeMap 2 subtrees , 0 ambiguous [metablob 0x1,       2 dirs]
10106     -2> 2021-12-31T21:08:02.861+0800 1461ab3b1700  4 mds.0.server handle_client_request client_request(client.4217:35 link #0x10000000000/file1 #      0x10000000cd7 2021-12-31T21:08:02.861659+0800 caller_uid=0, caller_gid=0{0,}) v5
10107     -1> 2021-12-31T21:08:02.889+0800 1461a9ba5700 -1 /data/ceph/src/mds/MDCache.cc: In function 'void MDCache::predirty_journal_parents(MutationR      ef, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)' thread 1461a9ba5700 time 2021-12-31T21:08:02.881576+0800
10108 /data/ceph/src/mds/MDCache.cc: 2168: FAILED ceph_assert(mut->is_wrlocked(&pin->filelock))
10109 
10110  ceph version 17.0.0-9829-g20d7b3fb38d (20d7b3fb38dec666cd6076841f72374f7aa945d2) quincy (dev)
10111  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x1461b099b929]
10112  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x1461b099bb5b]
10113  3: (MDCache::predirty_journal_parents(boost::intrusive_ptr<MutationImpl>, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0x5db) [0x55832711649d      ]
10114  4: (Server::_link_local(boost::intrusive_ptr<MDRequestImpl>&, CDentry*, CInode*, SnapRealm*)+0x5b6) [0x55832707cd9e]
10115  5: (Server::handle_client_link(boost::intrusive_ptr<MDRequestImpl>&)+0x1356) [0x55832707fc68]
10116  6: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x13e9) [0x5583270afa97]                                              
10117  7: (MDCache::dispatch_request(boost::intrusive_ptr<MDRequestImpl>&)+0x33) [0x55832718d4b3]
10118  8: (C_MDS_RetryRequest::finish(int)+0x2c) [0x55832718d5fa]
10119  9: (MDSContext::complete(int)+0x7e) [0x55832734ac22]
10120  10: (MDSCacheObject::finish_waiting(unsigned long, int)+0xd7a) [0x5583273780ae]
10121  11: (SimpleLock::finish_waiters(unsigned long, int)+0x26) [0x55832722100e]
10122  12: (Locker::eval_gather(SimpleLock*, bool, bool*, std::vector<MDSContext*, std::allocator<MDSContext*> >*)+0x16ff) [0x55832720a04f]
10123  13: (Locker::_finish_xlock(SimpleLock*, client_t, bool*)+0xbd) [0x55832720a30f]
10124  14: (Locker::xlock_finish(std::_Rb_tree_const_iterator<MutationImpl::LockOp> const&, MutationImpl*, bool*)+0x4fa) [0x558327210bb4]
10125  15: (Locker::_drop_locks(MutationImpl*, std::set<CInode*, std::less<CInode*>, std::allocator<CInode*> >*, bool)+0x96) [0x558327210c92]
10126  16: (Locker::drop_non_rdlocks(MutationImpl*, std::set<CInode*, std::less<CInode*>, std::allocator<CInode*> >*)+0x5a) [0x5583272125a8]
10127  17: (MDCache::request_drop_non_rdlocks(boost::intrusive_ptr<MDRequestImpl>&)+0x26) [0x5583271325da]
10128  18: (Server::reply_client_request(boost::intrusive_ptr<MDRequestImpl>&, boost::intrusive_ptr<MClientReply> const&)+0x3a1) [0x55832705dae3]
10129  19: (Server::respond_to_request(boost::intrusive_ptr<MDRequestImpl>&, int)+0x295) [0x55832705eb9d]
10130  20: (Server::_unlink_local_finish(boost::intrusive_ptr<MDRequestImpl>&, CDentry*, CDentry*, unsigned long)+0x597) [0x558327084c8b]
10131  21: (C_MDS_unlink_local_finish::finish(int)+0x24) [0x5583270fad4c]
10132  22: (MDSContext::complete(int)+0x7e) [0x55832734ac22]
10133  23: (MDSIOContextBase::complete(int)+0x19fa) [0x55832734ccd4]
10134  24: (MDSLogContextBase::complete(int)+0x47) [0x55832734d02b]
10135  25: (Finisher::finisher_thread_entry()+0x116b) [0x1461b0916225]
10136  26: (Finisher::FinisherThread::entry()+0xd) [0x558327022a55]
10137  27: (Thread::entry_wrapper()+0x3f) [0x1461b096e497]
10138  28: (Thread::_entry_func(void*)+0x9) [0x1461b096e4af]
10139  29: /lib64/libpthread.so.0(+0x82de) [0x1461af5a82de]
10140  30: clone()
10141 
10142      0> 2021-12-31T21:08:02.899+0800 1461a9ba5700 -1 *** Caught signal (Aborted) **
10143  in thread 1461a9ba5700 thread_name:MR_Finisher
10144 


Related issues

Copied to CephFS - Backport #53864: pacific: mds: FAILED ceph_assert(mut->is_wrlocked(&pin->filelock)) Resolved
Copied to CephFS - Backport #53865: octopus: mds: FAILED ceph_assert(mut->is_wrlocked(&pin->filelock)) Resolved

History

#1 Updated by Xiubo Li about 2 years ago

  • Status changed from New to In Progress

#2 Updated by Xiubo Li about 2 years ago

This is reproducable very easily by using the following scripts in two different teriminal:

$ while [ 1 ]; do rm -rf /mnt/cephfs/mydir/file1; ln /mnt/cephfs/mydir/file0 /mnt/cephfs/mydir/file1; echo '---------'; done

$ while [ 1 ]; do touch /mnt/kcephfs/mydir/file0; rm -rf /mnt/kcephfs/mydir/file0; echo "----"; done

#3 Updated by Xiubo Li about 2 years ago

  • Priority changed from Normal to High

#4 Updated by Xiubo Li about 2 years ago

  • Status changed from In Progress to Fix Under Review
  • Backport changed from pacific to pacific,octopus
  • Pull request ID set to 44451

#5 Updated by Venky Shankar about 2 years ago

  • Status changed from Fix Under Review to Pending Backport

#6 Updated by Backport Bot about 2 years ago

  • Copied to Backport #53864: pacific: mds: FAILED ceph_assert(mut->is_wrlocked(&pin->filelock)) added

#7 Updated by Backport Bot about 2 years ago

  • Copied to Backport #53865: octopus: mds: FAILED ceph_assert(mut->is_wrlocked(&pin->filelock)) added

#8 Updated by Xiubo Li about 2 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF