Bug #17656
Updated by Haomai Wang over 7 years ago
background: 背景： 因特殊的环境导致视频M3U8格式转码问题，采用cephfs 方式存储，前端采用NGX 进行视频转码到CEPH ，转码完成之后通知CDN 厂商预取视频文件，因网宿采用的是分片预取视频，导致NGINX 并非亮大(一分钟好几万个purge）导致NGINX 负载高，CEPH 告警！ we use cephfs as CDN backend, when CDN vendor prefetch video files in cephfs, it will cause high concurrent purge ops which cause ceph warning logs: 错误如下： 2016-10-21 09:12:58.505854 mds.0 192.168.194.118:6800/114274 14182 : cluster [WRN] 46 slow requests, 5 included below; oldest blocked for > 196.557580 secs 2016-10-21 09:12:58.505861 mds.0 192.168.194.118:6800/114274 14183 : cluster [WRN] slow request 121.856035 seconds old, received at 2016-10-21 09:10:56.649713: client_request(client.6137540:145761 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:10:56.656620) currently failed to rdlock, waiting 2016-10-21 09:12:58.505865 mds.0 192.168.194.118:6800/114274 14184 : cluster [WRN] slow request 121.731314 seconds old, received at 2016-10-21 09:10:56.774434: client_request(client.6137540:145762 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:10:56.781622) currently failed to rdlock, waiting 2016-10-21 09:12:58.505869 mds.0 192.168.194.118:6800/114274 14185 : cluster [WRN] slow request 121.145797 seconds old, received at 2016-10-21 09:10:57.359951: client_request(client.3759987:1135738 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:10:57.364154) currently failed to rdlock, waiting 2016-10-21 09:12:58.505873 mds.0 192.168.194.118:6800/114274 14186 : cluster [WRN] slow request 33.414109 seconds old, received at 2016-10-21 09:12:25.091639: client_request(client.6137540:145782 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:12:25.099527) currently failed to rdlock, waiting 2016-10-21 09:12:58.505877 mds.0 192.168.194.118:6800/114274 14187 : cluster [WRN] slow request 32.263839 seconds old, received at 2016-10-21 09:12:26.241909: client_request(client.3759987:1135763 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:12:26.245974) currently failed to rdlock, waiting 2016-10-21 09:13:03.505931 mds.0 192.168.194.118:6800/114274 14188 : cluster [WRN] 46 slow requests, 3 included below; oldest blocked for > 201.557676 secs 2016-10-21 09:13:03.505936 mds.0 192.168.194.118:6800/114274 14189 : cluster [WRN] slow request 124.578078 seconds old, received at 2016-10-21 09:10:58.927767: client_request(client.6137540:145763 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:10:58.934669) currently failed to rdlock, waiting 2016-10-21 09:13:03.505940 mds.0 192.168.194.118:6800/114274 14190 : cluster [WRN] slow request 130.552917 seconds old, received at 2016-10-21 09:10:52.952927: client_request(client.3759987:1135732 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:10:52.956163) currently failed to rdlock, waiting 2016-10-21 09:13:03.505943 mds.0 192.168.194.118:6800/114274 14191 : cluster [WRN] slow request 129.566257 seconds old, received at 2016-10-21 09:10:53.939588: client_request(client.6137540:145755 getattr pAsLsXsFs #1000001afb6 2016-10-21 09:10:53.946561) currently failed to rdlock, waiting