Project

General

Profile

Actions
Copy link

Bug #17656

open

cephfs: high concurrent causing slow request

Added by william sheng over 7 years ago. Updated about 7 years ago.

Status:
Need More Info
Priority:
Low
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
getattr pAsLsXsFs
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

background:

we use cephfs as CDN backend, when CDN vendor prefetch video files in cephfs, it will cause high concurrent purge ops which cause ceph warning

logs:
2016-10-21 09:12:58.505854 mds.0 192.168.194.118:6800/114274 14182 : cluster [WRN] 46 slow requests, 5 included below; oldest blocked for > 196.557580 secs
2016-10-21 09:12:58.505861 mds.0 192.168.194.118:6800/114274 14183 : cluster [WRN] slow request 121.856035 seconds old, received at 2016-10-21 09:10:56.649713: client_request(client.6137540:145761 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:10:56.656620) currently failed to rdlock, waiting
2016-10-21 09:12:58.505865 mds.0 192.168.194.118:6800/114274 14184 : cluster [WRN] slow request 121.731314 seconds old, received at 2016-10-21 09:10:56.774434: client_request(client.6137540:145762 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:10:56.781622) currently failed to rdlock, waiting
2016-10-21 09:12:58.505869 mds.0 192.168.194.118:6800/114274 14185 : cluster [WRN] slow request 121.145797 seconds old, received at 2016-10-21 09:10:57.359951: client_request(client.3759987:1135738 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:10:57.364154) currently failed to rdlock, waiting
2016-10-21 09:12:58.505873 mds.0 192.168.194.118:6800/114274 14186 : cluster [WRN] slow request 33.414109 seconds old, received at 2016-10-21 09:12:25.091639: client_request(client.6137540:145782 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:12:25.099527) currently failed to rdlock, waiting
2016-10-21 09:12:58.505877 mds.0 192.168.194.118:6800/114274 14187 : cluster [WRN] slow request 32.263839 seconds old, received at 2016-10-21 09:12:26.241909: client_request(client.3759987:1135763 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:12:26.245974) currently failed to rdlock, waiting
2016-10-21 09:13:03.505931 mds.0 192.168.194.118:6800/114274 14188 : cluster [WRN] 46 slow requests, 3 included below; oldest blocked for > 201.557676 secs
2016-10-21 09:13:03.505936 mds.0 192.168.194.118:6800/114274 14189 : cluster [WRN] slow request 124.578078 seconds old, received at 2016-10-21 09:10:58.927767: client_request(client.6137540:145763 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:10:58.934669) currently failed to rdlock, waiting
2016-10-21 09:13:03.505940 mds.0 192.168.194.118:6800/114274 14190 : cluster [WRN] slow request 130.552917 seconds old, received at 2016-10-21 09:10:52.952927: client_request(client.3759987:1135732 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:10:52.956163) currently failed to rdlock, waiting
2016-10-21 09:13:03.505943 mds.0 192.168.194.118:6800/114274 14191 : cluster [WRN] slow request 129.566257 seconds old, received at 2016-10-21 09:10:53.939588: client_request(client.6137540:145755 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:10:53.946561) currently failed to rdlock, waiting

Files

Download all files
node1.rar (730 KB) node1.rar jichao sun, 03/02/2017 11:51 AM
node2.rar (798 KB) node2.rar jichao sun, 03/02/2017 11:51 AM
node3.rar (811 KB) node3.rar jichao sun, 03/02/2017 11:51 AM
node4.rar (820 KB) node4.rar jichao sun, 03/02/2017 11:51 AM
node5.rar (808 KB) node5.rar jichao sun, 03/02/2017 11:51 AM
Actions
Copy link

Also available in: Atom PDF