Project

General

Profile

Actions

Bug #45078

open

mds lock always waiting stable after a bunch rename test

Added by Hikscale Team about 4 years ago. Updated almost 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Performance/Resource Usage
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
multimds
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I run a metadata bench test, which about 30,000 files rename test in one directory, and always got mds slow request about wrlock failed message.
And when we got mds slow request, the tested directory hangs and never recover even reboot all mds.

My test ceph environment is 14.2.8, has 3 active mds, deployed on CentOS 7.5.

The test Calready attached, which test metadata likes below:
In a directory A includes 30,000 files about 100MB~4GB randomly.
The test will sort all files by name, and place a class of file in new created subdir of directory A.

I dumped directory A inode in all of them. The inode log attached.

In mds log, we see export operation, so we config mds argument(mds_bal_split_xxx/mds_bal_fragment_xxx) to disable dir fragment.
After that, we run the metadata bench scripts, and the problem do not reprouducted.

Through the mds log attached, we found the inest lock is always waiting stable. Maybe there is something wrong on rename lock operations?


Files

dump_inode_on_rank0.txt (4.42 KB) dump_inode_on_rank0.txt Hikscale Team, 04/14/2020 06:41 AM
dump_inode_on_rank1.txt (3.74 KB) dump_inode_on_rank1.txt Hikscale Team, 04/14/2020 06:41 AM
md_bench_test.sh (281 Bytes) md_bench_test.sh Hikscale Team, 04/14/2020 06:41 AM
dump_inode_on_rank2.txt (4.1 KB) dump_inode_on_rank2.txt Hikscale Team, 04/14/2020 06:42 AM
mds.2.log.txt (34.6 KB) mds.2.log.txt Hikscale Team, 04/14/2020 06:43 AM
mds.0.log.txt (738 KB) mds.0.log.txt Hikscale Team, 04/14/2020 06:56 AM
Actions #2

Updated by Patrick Donnelly almost 4 years ago

  • Target version deleted (v14.2.8)
  • Affected Versions v14.2.8 added
Actions

Also available in: Atom PDF