Ceph » rgw

Hi

a ceph cluster here has been reporting blocked ops - when checking the OSD logs they report "currently waiting for scrub" - A restart of the OSD seems to overcome the issue in the short term, from the logs this impacts on both RGW+RBD and has been across 10's of OSD's, is there a way to track this down further? {ceph version 0.94.2}

The log samples are below:

2015-08-30 11:07:10.094473 7fcf71b36700 0 log_channel(cluster) log [WRN] : 15 slow requests, 1 included below; oldest blocked for > 16788.518542 secs
2015-08-30 11:07:10.094489 7fcf71b36700 0 log_channel(cluster) log [WRN] : slow request 15360.412949 seconds old, received at 2015-08-30 06:51:09.681469: osd_op(client.16410475.0:64466956 default.290071.4_01/2015/08/27/01.ZIP [create 0~0 [excl],setxattr user.rgw.idtag (25),writefull 0~0,setxattr user.rgw.manifest (403),setxattr user.rgw.acl (135),setxattr user.rgw.etag (33),setxattr user.rgw.x-amz-date (30)] 14.465004a7 ondisk+write+known_if_redirected e166614) currently waiting for scrub

2015-08-29 19:42:35.841226 7fa386997700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 7680.053142 secs
2015-08-29 19:42:35.841243 7fa386997700 0 log_channel(cluster) log [WRN] : slow request 7680.053142 seconds old, received at 2015-08-29 17:34:35.788038: osd_op(client.16900032.0:1515135 rbd_data.9583346c638121.000000000000092c [stat,set-alloc-hint object_size 8388608 write_size 8388608,write 3637248~61440] 4.89641761 snapc 193=[193] ack+ondisk+write+known_if_redirected e166602) currently waiting for scrub
2015-08-29 19:45:16.191408 7fa3420b4700 0 -- 172.28.196.222:6869/224634