Project

General

Profile

Bug #18519

speed up readdir by skipping unwanted dn

Added by Xiaoxi Chen about 7 years ago. Updated about 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
jewel, kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

we hit MDS CPU bottleneck (100% on one core as it is single thread) in our cephFS production enviroment.

Troubleshooting showing one applications are calling readdir often, what is worth, the dir has 130K files. Profiling showing on average each readdir call take ~20ms to finish in this scale(#files) and waste a significant time(and CPU!) on skipping unwanted dentries.

Take this request as example: 10 ms was spent on skipping the dentries and 7 ms was spent on encoding the wanted 1024 dentries. This patch addressed the 10ms and attempt to minimize it.

2017-01-09 19:49:03.023878 7ff8d74ce700 10 mds.0.server snapid head

[Iterating and skipping all dentry < offset]
https://github.com/ceph/ceph/blob/v10.2.2/src/mds/Server.cc#L3379

2017-01-09 19:49:03.033836 7ff8d74ce700 10 mds.0.cache.ino(100000edea3) encode_inodestat issuing pAsLsXsFscr seq 1867
[Encoding inodes to reply message, 1024 inodes in total , +10.836ms]
2017-01-09 19:49:03.040745 7ff8d74ce700 10 mds.0.cache.ino(100000eb3e7) encode_inodestat issuing pAsLsXsFscr seq 1867
……

[Finished encoding, reply to client,total size ~300KB + 17.752ms]
2017-01-09 19:49:03.040752 7ff8d74ce700 10 mds.0.server reply to client_request(client.26741659:858764 readdir #100000cbe8d lvs3b02c-cb62.stratus.lvs.ebay.com.pem 2017-01-09 19:49:02.991586) v2 readdir num=1024 bytes=310510 end=0 complete=0


Related issues

Copied to CephFS - Backport #18520: jewel: speed up readdir by skipping unwanted dn Resolved
Copied to CephFS - Backport #18531: kraken: speed up readdir by skipping unwanted dn Resolved

History

#2 Updated by Xiaoxi Chen about 7 years ago

  • Copied to Backport #18520: jewel: speed up readdir by skipping unwanted dn added

#3 Updated by Xiaoxi Chen about 7 years ago

  • Backport set to jewel, kraken

#4 Updated by Xiaoxi Chen about 7 years ago

  • Copied to Backport #18531: kraken: speed up readdir by skipping unwanted dn added

#5 Updated by Nathan Cutler about 7 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF