Ceph

Ceph version is:
ceph version 0.56.2 (586538e22afba85c59beda49789ec42024e7a061)

Partial logfile included

2013-04-06 20:24:32.191724 7f36241ac700  0 log [WRN] : slow request 961.004978 seconds old, received at 2013-04-06 20:08:31.186695: osd_op(client.11689.1:430921688 rb.0.2b47.238e1f29.000000000659 [write 3432448~4096] 2.fc7d1e91 RETRY) currently reached pg
2013-04-06 20:24:32.191727 7f36241ac700  0 log [WRN] : slow request 961.004829 seconds old, received at 2013-04-06 20:08:31.186844: osd_op(client.11689.1:430926749 rb.0.376f.238e1f29.00000000877f [write 3411968~8192] 2.503ed691 RETRY) currently reached pg
2013-04-06 20:24:32.191729 7f36241ac700  0 log [WRN] : slow request 960.760384 seconds old, received at 2013-04-06 20:08:31.431289: osd_op(client.11689.1:430927698 rb.0.376f.238e1f29.00000000877f [write 3477504~8192] 2.503ed691 RETRY) currently reached pg
2013-04-06 20:24:32.191731 7f36241ac700  0 log [WRN] : slow request 480.259587 seconds old, received at 2013-04-06 20:16:31.932086: osd_op(client.11689.1:430921219 rb.0.2b11.2ae8944a.000000000b23 [write 2707456~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:32.191733 7f36241ac700  0 log [WRN] : slow request 480.259056 seconds old, received at 2013-04-06 20:16:31.932617: osd_op(client.11689.1:430921220 rb.0.2b11.2ae8944a.000000000b23 [write 2740224~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:33.191859 7f36241ac700  0 log [WRN] : 129 slow requests, 6 included below; oldest blocked for > 1322.872554 secs
2013-04-06 20:24:33.191865 7f36241ac700  0 log [WRN] : slow request 481.258547 seconds old, received at 2013-04-06 20:16:31.933263: osd_op(client.11689.1:430921221 rb.0.2b11.2ae8944a.000000000b23 [write 3018752~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:33.191869 7f36241ac700  0 log [WRN] : slow request 481.257981 seconds old, received at 2013-04-06 20:16:31.933829: osd_op(client.11689.1:430921222 rb.0.2b11.2ae8944a.000000000b23 [write 3084288~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:33.191874 7f36241ac700  0 log [WRN] : slow request 481.257458 seconds old, received at 2013-04-06 20:16:31.934352: osd_op(client.11689.1:430921223 rb.0.2b11.2ae8944a.000000000b23 [write 3133440~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:33.191881 7f36241ac700  0 log [WRN] : slow request 481.257072 seconds old, received at 2013-04-06 20:16:31.934738: osd_op(client.11689.1:430921688 rb.0.2b47.238e1f29.000000000659 [write 3432448~4096] 2.fc7d1e91 RETRY) currently reached pg
2013-04-06 20:24:33.191885 7f36241ac700  0 log [WRN] : slow request 481.256979 seconds old, received at 2013-04-06 20:16:31.934831: osd_op(client.11689.1:430926749 rb.0.376f.238e1f29.00000000877f [write 3411968~8192] 2.503ed691 RETRY) currently reached pg
2013-04-06 20:24:34.192013 7f36241ac700  0 log [WRN] : 165 slow requests, 6 included below; oldest blocked for > 1323.872706 secs
2013-04-06 20:24:34.192020 7f36241ac700  0 log [WRN] : slow request 482.256898 seconds old, received at 2013-04-06 20:16:31.935064: osd_op(client.11689.1:430927698 rb.0.376f.238e1f29.00000000877f [write 3477504~8192] 2.503ed691 RETRY) currently reached pg
2013-04-06 20:24:34.192025 7f36241ac700  0 log [WRN] : slow request 482.256689 seconds old, received at 2013-04-06 20:16:31.935273: osd_op(client.11689.1:430933595 rb.0.376c.238e1f29.000000000005 [write 12288~4096] 2.f048f091 RETRY) currently reached pg
2013-04-06 20:24:34.192033 7f36241ac700  0 log [WRN] : slow request 482.256533 seconds old, received at 2013-04-06 20:16:31.935429: osd_op(client.11689.1:430934521 rb.0.376c.238e1f29.000000000005 [write 16384~16384] 2.f048f091 RETRY) currently reached pg
2013-04-06 20:24:34.192036 7f36241ac700  0 log [WRN] : slow request 241.886051 seconds old, received at 2013-04-06 20:20:32.305911: osd_op(client.11689.1:430921219 rb.0.2b11.2ae8944a.000000000b23 [write 2707456~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:34.192040 7f36241ac700  0 log [WRN] : slow request 241.885188 seconds old, received at 2013-04-06 20:20:32.306774: osd_op(client.11689.1:430921220 rb.0.2b11.2ae8944a.000000000b23 [write 2740224~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:35.192164 7f36241ac700  0 log [WRN] : 170 slow requests, 6 included below; oldest blocked for > 1324.872859 secs
2013-04-06 20:24:35.192169 7f36241ac700  0 log [WRN] : slow request 242.884965 seconds old, received at 2013-04-06 20:20:32.307150: osd_op(client.11689.1:430921221 rb.0.2b11.2ae8944a.000000000b23 [write 3018752~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:35.192171 7f36241ac700  0 log [WRN] : slow request 242.884065 seconds old, received at 2013-04-06 20:20:32.308050: osd_op(client.11689.1:430921222 rb.0.2b11.2ae8944a.000000000b23 [write 3084288~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:35.192173 7f36241ac700  0 log [WRN] : slow request 242.883487 seconds old, received at 2013-04-06 20:20:32.308628: osd_op(client.11689.1:430921223 rb.0.2b11.2ae8944a.000000000b23 [write 3133440~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:35.192175 7f36241ac700  0 log [WRN] : slow request 242.883110 seconds old, received at 2013-04-06 20:20:32.309005: osd_op(client.11689.1:430921688 rb.0.2b47.238e1f29.000000000659 [write 3432448~4096] 2.fc7d1e91 RETRY) currently reached pg
2013-04-06 20:24:35.192177 7f36241ac700  0 log [WRN] : slow request 242.883021 seconds old, received at 2013-04-06 20:20:32.309094: osd_op(client.11689.1:430926749 rb.0.376f.238e1f29.00000000877f [write 3411968~8192] 2.503ed691 RETRY) currently reached pg
2013-04-06 20:24:36.192305 7f36241ac700  0 log [WRN] : 230 slow requests, 4 included below; oldest blocked for > 1325.873006 secs
2013-04-06 20:24:36.192310 7f36241ac700  0 log [WRN] : slow request 243.882966 seconds old, received at 2013-04-06 20:20:32.309296: osd_op(client.11689.1:430927698 rb.0.376f.238e1f29.00000000877f [write 3477504~8192] 2.503ed691 RETRY) currently reached pg
2013-04-06 20:24:36.192312 7f36241ac700  0 log [WRN] : slow request 243.882699 seconds old, received at 2013-04-06 20:20:32.309563: osd_op(client.11689.1:430933595 rb.0.376c.238e1f29.000000000005 [write 12288~4096] 2.f048f091 RETRY) currently reached pg
2013-04-06 20:24:36.192315 7f36241ac700  0 log [WRN] : slow request 243.882384 seconds old, received at 2013-04-06 20:20:32.309878: osd_op(client.11689.1:430934521 rb.0.376c.238e1f29.000000000005 [write 16384~16384] 2.f048f091 RETRY) currently reached pg
2013-04-06 20:24:36.192317 7f36241ac700  0 log [WRN] : slow request 243.881706 seconds old, received at 2013-04-06 20:20:32.310556: osd_op(client.11689.1:430954484 rb.0.2cf8.2ae8944a.0000000008ec [read 1179648~12288] 2.8b9f1e91 RETRY) currently reached pg
2013-04-06 20:24:38.192556 7f36241ac700  0 log [WRN] : 202 slow requests, 6 included below; oldest blocked for > 1327.873253 secs
2013-04-06 20:24:38.192560 7f36241ac700  0 log [WRN] : slow request 120.691588 seconds old, received at 2013-04-06 20:22:37.500921: osd_op(client.11689.1:430921219 rb.0.2b11.2ae8944a.000000000b23 [write 2707456~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:38.192563 7f36241ac700  0 log [WRN] : slow request 120.691125 seconds old, received at 2013-04-06 20:22:37.501384: osd_op(client.11689.1:430921220 rb.0.2b11.2ae8944a.000000000b23 [write 2740224~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:38.192565 7f36241ac700  0 log [WRN] : slow request 120.690351 seconds old, received at 2013-04-06 20:22:37.502158: osd_op(client.11689.1:430921221 rb.0.2b11.2ae8944a.000000000b23 [write 3018752~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:38.192567 7f36241ac700  0 log [WRN] : slow request 120.689772 seconds old, received at 2013-04-06 20:22:37.502737: osd_op(client.11689.1:430921222 rb.0.2b11.2ae8944a.000000000b23 [write 3084288~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:38.192569 7f36241ac700  0 log [WRN] : slow request 120.689315 seconds old, received at 2013-04-06 20:22:37.503194: osd_op(client.11689.1:430921223 rb.0.2b11.2ae8944a.000000000b23 [write 3133440~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:39.192698 7f36241ac700  0 log [WRN] : 179 slow requests, 6 included below; oldest blocked for > 1328.873390 secs
2013-04-06 20:24:39.192704 7f36241ac700  0 log [WRN] : slow request 240.919513 seconds old, received at 2013-04-06 20:20:38.273133: osd_op(client.11689.1:430960718 rb.0.2cd1.238e1f29.000000000bdd [write 1585152~4096] 2.8cafbe91) currently reached pg
2013-04-06 20:24:39.192710 7f36241ac700  0 log [WRN] : slow request 240.919471 seconds old, received at 2013-04-06 20:20:38.273175: osd_op(client.11689.1:430960719 rb.0.2cd1.238e1f29.000000000bdd [write 2019328~4096] 2.8cafbe91) currently reached pg
2013-04-06 20:24:39.192713 7f36241ac700  0 log [WRN] : slow request 240.916626 seconds old, received at 2013-04-06 20:20:38.276020: osd_op(client.11689.1:430960720 rb.0.2cd1.238e1f29.000000000bdd [write 2551808~4096] 2.8cafbe91) currently reached pg
2013-04-06 20:24:39.192720 7f36241ac700  0 log [WRN] : slow request 240.916560 seconds old, received at 2013-04-06 20:20:38.276086: osd_op(client.11689.1:430960721 rb.0.2cd1.238e1f29.000000000bdd [write 2555904~4096] 2.8cafbe91) currently reached pg
2013-04-06 20:24:39.192723 7f36241ac700  0 log [WRN] : slow request 240.916508 seconds old, received at 2013-04-06 20:20:38.276138: osd_op(client.11689.1:430960722 rb.0.2cd1.238e1f29.000000000bdd [write 2560000~12288] 2.8cafbe91) currently reached pg
2013-04-06 20:24:40.192855 7f36241ac700  0 log [WRN] : 206 slow requests, 6 included below; oldest blocked for > 1329.873551 secs
2013-04-06 20:24:40.192859 7f36241ac700  0 log [WRN] : slow request 241.915836 seconds old, received at 2013-04-06 20:20:38.276971: osd_op(client.11689.1:430960723 rb.0.2cd1.238e1f29.000000000bdd [write 2547712~4096] 2.8cafbe91) currently reached pg
2013-04-06 20:24:40.192862 7f36241ac700  0 log [WRN] : slow request 122.689217 seconds old, received at 2013-04-06 20:22:37.503590: osd_op(client.11689.1:430921688 rb.0.2b47.238e1f29.000000000659 [write 3432448~4096] 2.fc7d1e91 RETRY) currently reached pg
2013-04-06 20:24:40.192864 7f36241ac700  0 log [WRN] : slow request 122.688989 seconds old, received at 2013-04-06 20:22:37.503818: osd_op(client.11689.1:430926749 rb.0.376f.238e1f29.00000000877f [write 3411968~8192] 2.503ed691 RETRY) currently reached pg
2013-04-06 20:24:40.192866 7f36241ac700  0 log [WRN] : slow request 122.688834 seconds old, received at 2013-04-06 20:22:37.503973: osd_op(client.11689.1:430927698 rb.0.376f.238e1f29.00000000877f [write 3477504~8192] 2.503ed691 RETRY) currently reached pg
2013-04-06 20:24:40.192868 7f36241ac700  0 log [WRN] : slow request 122.688666 seconds old, received at 2013-04-06 20:22:37.504141: osd_op(client.11689.1:430933595 rb.0.376c.238e1f29.000000000005 [write 12288~4096] 2.f048f091 RETRY) currently reached pg
2013-04-06 20:24:41.192999 7f36241ac700  0 log [WRN] : 211 slow requests, 6 included below; oldest blocked for > 1330.873688 secs
2013-04-06 20:24:41.193005 7f36241ac700  0 log [WRN] : slow request 123.688672 seconds old, received at 2013-04-06 20:22:37.504272: osd_op(client.11689.1:430934521 rb.0.376c.238e1f29.000000000005 [write 16384~16384] 2.f048f091 RETRY) currently reached pg
2013-04-06 20:24:41.193009 7f36241ac700  0 log [WRN] : slow request 123.688342 seconds old, received at 2013-04-06 20:22:37.504602: osd_op(client.11689.1:430954484 rb.0.2cf8.2ae8944a.0000000008ec [read 1179648~12288] 2.8b9f1e91 RETRY) currently reached pg
2013-04-06 20:24:41.193013 7f36241ac700  0 log [WRN] : slow request 123.688329 seconds old, received at 2013-04-06 20:22:37.504615: osd_op(client.11689.1:430960718 rb.0.2cd1.238e1f29.000000000bdd [write 1585152~4096] 2.8cafbe91 RETRY) currently reached pg
2013-04-06 20:24:41.193016 7f36241ac700  0 log [WRN] : slow request 123.688273 seconds old, received at 2013-04-06 20:22:37.504671: osd_op(client.11689.1:430960719 rb.0.2cd1.238e1f29.000000000bdd [write 2019328~4096] 2.8cafbe91 RETRY) currently reached pg
2013-04-06 20:24:41.193019 7f36241ac700  0 log [WRN] : slow request 123.688241 seconds old, received at 2013-04-06 20:22:37.504703: osd_op(client.11689.1:430960720 rb.0.2cd1.238e1f29.000000000bdd [write 2551808~4096] 2.8cafbe91 RETRY) currently reached pg
2013-04-06 20:24:42.193145 7f36241ac700  0 log [WRN] : 216 slow requests, 6 included below; oldest blocked for > 1331.873850 secs
2013-04-06 20:24:42.193150 7f36241ac700  0 log [WRN] : slow request 124.688280 seconds old, received at 2013-04-06 20:22:37.504826: osd_op(client.11689.1:430960721 rb.0.2cd1.238e1f29.000000000bdd [write 2555904~4096] 2.8cafbe91 RETRY) currently reached pg
2013-04-06 20:24:42.193155 7f36241ac700  0 log [WRN] : slow request 124.688229 seconds old, received at 2013-04-06 20:22:37.504877: osd_op(client.11689.1:430960722 rb.0.2cd1.238e1f29.000000000bdd [write 2560000~12288] 2.8cafbe91 RETRY) currently reached pg
2013-04-06 20:24:42.193159 7f36241ac700  0 log [WRN] : slow request 124.687950 seconds old, received at 2013-04-06 20:22:37.505156: osd_op(client.11689.1:430960723 rb.0.2cd1.238e1f29.000000000bdd [write 2547712~4096] 2.8cafbe91 RETRY) currently reached pg
2013-04-06 20:24:42.193162 7f36241ac700  0 log [WRN] : slow request 64.599147 seconds old, received at 2013-04-06 20:23:37.593959: osd_op(client.11689.1:430921219 rb.0.2b11.2ae8944a.000000000b23 [write 2707456~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:42.193166 7f36241ac700  0 log [WRN] : slow request 64.598611 seconds old, received at 2013-04-06 20:23:37.594495: osd_op(client.11689.1:430921220 rb.0.2b11.2ae8944a.000000000b23 [write 2740224~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:43.193310 7f36241ac700  0 log [WRN] : 221 slow requests, 6 included below; oldest blocked for > 1332.874000 secs
2013-04-06 20:24:43.193314 7f36241ac700  0 log [WRN] : slow request 65.598300 seconds old, received at 2013-04-06 20:23:37.594956: osd_op(client.11689.1:430921221 rb.0.2b11.2ae8944a.000000000b23 [write 3018752~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:43.193317 7f36241ac700  0 log [WRN] : slow request 65.597978 seconds old, received at 2013-04-06 20:23:37.595278: osd_op(client.11689.1:430921222 rb.0.2b11.2ae8944a.000000000b23 [write 3084288~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:43.193319 7f36241ac700  0 log [WRN] : slow request 65.597528 seconds old, received at 2013-04-06 20:23:37.595728: osd_op(client.11689.1:430921223 rb.0.2b11.2ae8944a.000000000b23 [write 3133440~20480] 2.f66a7491 RETRY) currently reached pg
2013-04-06 20:24:43.193321 7f36241ac700  0 log [WRN] : slow request 65.597044 seconds old, received at 2013-04-06 20:23:37.596212: osd_op(client.11689.1:430921688 rb.0.2b47.238e1f29.000000000659 [write 3432448~4096] 2.fc7d1e91 RETRY) currently reached pg
2013-04-06 20:24:43.193323 7f36241ac700  0 log [WRN] : slow request 65.597003 seconds old, received at 2013-04-06 20:23:37.596253: osd_op(client.11689.1:430926749 rb.0.376f.238e1f29.00000000877f [write 3411968~8192] 2.503ed691 RETRY) currently reached pg
2013-04-06 20:24:44.193454 7f36241ac700  0 log [WRN] : 226 slow requests, 6 included below; oldest blocked for > 1333.874144 secs
2013-04-06 20:24:44.193459 7f36241ac700  0 log [WRN] : slow request 66.596852 seconds old, received at 2013-04-06 20:23:37.596548: osd_op(client.11689.1:430927698 rb.0.376f.238e1f29.00000000877f [write 3477504~8192] 2.503ed691 RETRY) currently reached pg
2013-04-06 20:24:44.193462 7f36241ac700  0 log [WRN] : slow request 66.596707 seconds old, received at 2013-04-06 20:23:37.596693: osd_op(client.11689.1:430933595 rb.0.376c.238e1f29.000000000005 [write 12288~4096] 2.f048f091 RETRY) currently reached pg
2013-04-06 20:24:44.193464 7f36241ac700  0 log [WRN] : slow request 66.596599 seconds old, received at 2013-04-06 20:23:37.596801: osd_op(client.11689.1:430934521 rb.0.376c.238e1f29.000000000005 [write 16384~16384] 2.f048f091 RETRY) currently reached pg
2013-04-06 20:24:44.193466 7f36241ac700  0 log [WRN] : slow request 66.595991 seconds old, received at 2013-04-06 20:23:37.597409: osd_op(client.11689.1:430954484 rb.0.2cf8.2ae8944a.0000000008ec [read 1179648~12288] 2.8b9f1e91 RETRY) currently reached pg
2013-04-06 20:24:44.193468 7f36241ac700  0 log [WRN] : slow request 66.595983 seconds old, received at 2013-04-06 20:23:37.597417: osd_op(client.11689.1:430960718 rb.0.2cd1.238e1f29.000000000bdd [write 1585152~4096] 2.8cafbe91 RETRY) currently reached pg
2013-04-06 20:24:45.193608 7f36241ac700  0 log [WRN] : 230 slow requests, 5 included below; oldest blocked for > 1334.874299 secs
2013-04-06 20:24:45.193613 7f36241ac700  0 log [WRN] : slow request 67.596068 seconds old, received at 2013-04-06 20:23:37.597487: osd_op(client.11689.1:430960719 rb.0.2cd1.238e1f29.000000000bdd [write 2019328~4096] 2.8cafbe91 RETRY) currently reached pg
2013-04-06 20:24:45.193615 7f36241ac700  0 log [WRN] : slow request 67.595914 seconds old, received at 2013-04-06 20:23:37.597641: osd_op(client.11689.1:430960720 rb.0.2cd1.238e1f29.000000000bdd [write 2551808~4096] 2.8cafbe91 RETRY) currently reached pg
2013-04-06 20:24:45.193617 7f36241ac700  0 log [WRN] : slow request 67.595695 seconds old, received at 2013-04-06 20:23:37.597860: osd_op(client.11689.1:430960721 rb.0.2cd1.238e1f29.000000000bdd [write 2555904~4096] 2.8cafbe91 RETRY) currently reached pg
2013-04-06 20:24:45.193620 7f36241ac700  0 log [WRN] : slow request 67.595579 seconds old, received at 2013-04-06 20:23:37.597976: osd_op(client.11689.1:430960722 rb.0.2cd1.238e1f29.000000000bdd [write 2560000~12288] 2.8cafbe91 RETRY) currently reached pg
2013-04-06 20:24:45.193622 7f36241ac700  0 log [WRN] : slow request 67.352079 seconds old, received at 2013-04-06 20:23:37.841476: osd_op(client.11689.1:430960723 rb.0.2cd1.238e1f29.000000000bdd [write 2547712~4096] 2.8cafbe91 RETRY) currently reached pg
2013-04-06 20:24:50.113105 7f2734d03780  0 filestore(/srv/ceph/osd0) mount WARNING: multiple ceph-osd daemons on the same host will be slow
2013-04-06 20:24:50.471914 7f2734d03780  0 filestore(/srv/ceph/osd0) mount WARNING: multiple ceph-osd daemons on the same host will be slow
2013-04-06 20:43:11.881009 7f272fc94700  0 log [WRN] : 5 slow requests, 5 included below; oldest blocked for > 30.087053 secs
2013-04-06 20:43:11.881014 7f272fc94700  0 log [WRN] : slow request 30.087053 seconds old, received at 2013-04-06 20:42:41.793913: osd_op(client.11689.1:431035225 rb.0.2b11.2ae8944a.000000000b23 [write 2707456~20480] 2.f66a7491) currently reached pg
2013-04-06 20:43:11.881017 7f272fc94700  0 log [WRN] : slow request 30.084432 seconds old, received at 2013-04-06 20:42:41.796534: osd_op(client.11689.1:431035226 rb.0.2b11.2ae8944a.000000000b23 [write 2740224~20480] 2.f66a7491) currently reached pg
2013-04-06 20:43:11.881020 7f272fc94700  0 log [WRN] : slow request 30.082510 seconds old, received at 2013-04-06 20:42:41.798456: osd_op(client.11689.1:431035227 rb.0.2b11.2ae8944a.000000000b23 [write 3018752~20480] 2.f66a7491) currently reached pg
2013-04-06 20:43:11.881022 7f272fc94700  0 log [WRN] : slow request 30.080215 seconds old, received at 2013-04-06 20:42:41.800751: osd_op(client.11689.1:431035228 rb.0.2b11.2ae8944a.000000000b23 [write 3084288~20480] 2.f66a7491) currently reached pg
2013-04-06 20:43:11.881023 7f272fc94700  0 log [WRN] : slow request 30.077425 seconds old, received at 2013-04-06 20:42:41.803541: osd_op(client.11689.1:431035229 rb.0.2b11.2ae8944a.000000000b23 [write 3133440~20480] 2.f66a7491) currently reached pg
2013-04-06 20:43:41.883794 7f272fc94700  0 log [WRN] : 5 slow requests, 5 included below; oldest blocked for > 60.089843 secs
2013-04-06 20:43:41.883799 7f272fc94700  0 log [WRN] : slow request 60.089843 seconds old, received at 2013-04-06 20:42:41.793913: osd_op(client.11689.1:431035225 rb.0.2b11.2ae8944a.000000000b23 [write 2707456~20480] 2.f66a7491) currently reached pg
2013-04-06 20:43:41.883802 7f272fc94700  0 log [WRN] : slow request 60.087222 seconds old, received at 2013-04-06 20:42:41.796534: osd_op(client.11689.1:431035226 rb.0.2b11.2ae8944a.000000000b23 [write 2740224~20480] 2.f66a7491) currently reached pg
2013-04-06 20:43:41.883805 7f272fc94700  0 log [WRN] : slow request 60.085300 seconds old, received at 2013-04-06 20:42:41.798456: osd_op(client.11689.1:431035227 rb.0.2b11.2ae8944a.000000000b23 [write 3018752~20480] 2.f66a7491) currently reached pg
2013-04-06 20:43:41.883807 7f272fc94700  0 log [WRN] : slow request 60.083005 seconds old, received at 2013-04-06 20:42:41.800751: osd_op(client.11689.1:431035228 rb.0.2b11.2ae8944a.000000000b23 [write 3084288~20480] 2.f66a7491) currently reached pg
2013-04-06 20:43:41.883809 7f272fc94700  0 log [WRN] : slow request 60.080215 seconds old, received at 2013-04-06 20:42:41.803541: osd_op(client.11689.1:431035229 rb.0.2b11.2ae8944a.000000000b23 [write 3133440~20480] 2.f66a7491) currently reached pg