|
2015-10-26 06:32:36.796274 7f552d120700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.244:6801/11164 pipe(0xe6a6d000 sd=120 :52504 s=2 pgs=2914 cs=5 l=0 c=0xe60247e0).fault with nothing to send, going to standby
|
|
2015-10-26 06:33:05.544705 7f54de21d700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.253:6804/21530 pipe(0xe2c5b000 sd=374 :6811 s=2 pgs=1 cs=1 l=0 c=0xd8c459c0).fault with nothing to send, going to standby
|
|
2015-10-26 06:33:20.103397 7f55417cf700 0 log_channel(cluster) log [WRN] : 7 slow requests, 1 included below; oldest blocked for > 15360.252048 secs
|
|
2015-10-26 06:33:20.103403 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 15360.252048 seconds old, received at 2015-10-26 02:17:19.851323: osd_op(client.31410008.0:144241 rbd_data.43aabb2ae8944a.0000000000000608 [set-alloc-hint object_size 4194304 write_size 4194304,write 171520~4096] 58.d1eb9844 ack+ondisk+write+known_if_redirected e90157) currently reached_pg
|
|
2015-10-26 06:36:29.438655 7f55331fb700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.239:6820/105498 pipe(0xdefc8000 sd=283 :59994 s=2 pgs=124 cs=7 l=0 c=0x14bbf6c60).fault with nothing to send, going to standby
|
|
2015-10-26 06:40:37.440401 7f54e57e9700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.242:6808/24245 pipe(0x17fa4000 sd=299 :53457 s=2 pgs=1376 cs=3 l=0 c=0xd6cb1020).fault with nothing to send, going to standby
|
|
2015-10-26 06:41:07.880549 7f55417cf700 0 log_channel(cluster) log [WRN] : 7 slow requests, 2 included below; oldest blocked for > 15828.029200 secs
|
|
2015-10-26 06:41:07.880553 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 7680.337449 seconds old, received at 2015-10-26 04:33:07.543074: osd_op(client.23210480.0:13447318 rb.0.40d9d0.238e1f29.000000000235 [sparse-read 2792960~118784] 58.5f199844 ack+read+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:41:07.880558 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 7680.337321 seconds old, received at 2015-10-26 04:33:07.543202: osd_op(client.23210480.0:13447319 rb.0.40d9d0.238e1f29.000000000235 [sparse-read 3067392~12288] 58.5f199844 ack+read+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:41:24.960439 7f551bb8b700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.243:6802/22989 pipe(0x10671000 sd=302 :34107 s=2 pgs=1864 cs=9 l=0 c=0xd8c43e40).fault with nothing to send, going to standby
|
|
2015-10-26 06:41:40.242776 7f55417cf700 0 log_channel(cluster) log [WRN] : 7 slow requests, 1 included below; oldest blocked for > 15860.391418 secs
|
|
2015-10-26 06:41:40.242816 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 15360.600666 seconds old, received at 2015-10-26 02:25:39.642075: osd_op(client.31409819.0:68838 rb.0.d8701.2ae8944a.000000000e7a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~8192] 58.72a1073e ack+ondisk+write+known_if_redirected e90284) currently reached_pg
|
|
2015-10-26 06:42:55.491529 7f54dd813700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.242:6808/24245 pipe(0x147ff4000 sd=294 :6811 s=0 pgs=0 cs=0 l=0 c=0xd6cae580).accept connect_seq 4 vs existing 3 state standby
|
|
2015-10-26 06:43:35.543843 7f550d410700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.253:6804/21530 pipe(0x127fa9000 sd=253 :6811 s=0 pgs=0 cs=0 l=0 c=0xe809ec00).accept connect_seq 2 vs existing 1 state standby
|
|
2015-10-26 06:45:34.221506 7f55417cf700 0 log_channel(cluster) log [WRN] : 10 slow requests, 3 included below; oldest blocked for > 16094.370135 secs
|
|
2015-10-26 06:45:34.221513 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.341634 seconds old, received at 2015-10-26 06:45:03.879824: osd_op(client.23194930.0:4774508 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~94208] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:45:34.221531 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.340779 seconds old, received at 2015-10-26 06:45:03.880680: osd_op(client.23194930.0:4774509 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~131072] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:45:34.221534 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.340523 seconds old, received at 2015-10-26 06:45:03.880935: osd_op(client.23194930.0:4774510 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~524288] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:46:04.218135 7f55417cf700 0 log_channel(cluster) log [WRN] : 10 slow requests, 3 included below; oldest blocked for > 16124.366783 secs
|
|
2015-10-26 06:46:04.218140 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 60.338282 seconds old, received at 2015-10-26 06:45:03.879824: osd_op(client.23194930.0:4774508 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~94208] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:46:04.218143 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 60.337426 seconds old, received at 2015-10-26 06:45:03.880680: osd_op(client.23194930.0:4774509 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~131072] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:46:04.218145 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 60.337171 seconds old, received at 2015-10-26 06:45:03.880935: osd_op(client.23194930.0:4774510 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~524288] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:47:03.898958 7f55417cf700 0 log_channel(cluster) log [WRN] : 10 slow requests, 3 included below; oldest blocked for > 16184.047580 secs
|
|
2015-10-26 06:47:03.898967 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 120.019079 seconds old, received at 2015-10-26 06:45:03.879824: osd_op(client.23194930.0:4774508 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~94208] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:47:03.898971 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 120.018224 seconds old, received at 2015-10-26 06:45:03.880680: osd_op(client.23194930.0:4774509 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~131072] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:47:03.898973 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 120.017968 seconds old, received at 2015-10-26 06:45:03.880935: osd_op(client.23194930.0:4774510 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~524288] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:47:52.912442 7f54ebebc700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.245:6820/23219 pipe(0x14a036000 sd=80 :54129 s=2 pgs=776 cs=5 l=0 c=0x26dfcc60).fault with nothing to send, going to standby
|
|
2015-10-26 06:49:04.135380 7f55417cf700 0 log_channel(cluster) log [WRN] : 10 slow requests, 3 included below; oldest blocked for > 16304.284028 secs
|
|
2015-10-26 06:49:04.135385 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 240.255527 seconds old, received at 2015-10-26 06:45:03.879824: osd_op(client.23194930.0:4774508 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~94208] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:49:04.135413 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 240.254671 seconds old, received at 2015-10-26 06:45:03.880680: osd_op(client.23194930.0:4774509 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~131072] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:49:04.135416 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 240.254416 seconds old, received at 2015-10-26 06:45:03.880935: osd_op(client.23194930.0:4774510 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~524288] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:49:47.563441 7f54dec27700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.253:6803/2983 pipe(0xfd6fe000 sd=337 :6811 s=2 pgs=1744 cs=5 l=0 c=0xe809fde0).fault with nothing to send, going to standby
|
|
2015-10-26 06:52:21.587426 7f550e040700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.244:6806/18405 pipe(0xd895f000 sd=29 :53599 s=2 pgs=2483 cs=9 l=0 c=0xe6023e40).fault with nothing to send, going to standby
|
|
2015-10-26 06:53:04.215934 7f55417cf700 0 log_channel(cluster) log [WRN] : 10 slow requests, 3 included below; oldest blocked for > 16544.364577 secs
|
|
2015-10-26 06:53:04.215940 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 480.336076 seconds old, received at 2015-10-26 06:45:03.879824: osd_op(client.23194930.0:4774508 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~94208] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:53:04.215944 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 480.335220 seconds old, received at 2015-10-26 06:45:03.880680: osd_op(client.23194930.0:4774509 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~131072] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:53:04.215947 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 480.334965 seconds old, received at 2015-10-26 06:45:03.880935: osd_op(client.23194930.0:4774510 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~524288] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 06:58:35.644886 7f550d410700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.253:6804/21530 pipe(0x127fa9000 sd=253 :6811 s=2 pgs=1620 cs=3 l=0 c=0xd8c459c0).fault with nothing to send, going to standby
|
|
2015-10-26 07:01:03.981731 7f55417cf700 0 log_channel(cluster) log [WRN] : 10 slow requests, 3 included below; oldest blocked for > 17024.130376 secs
|
|
2015-10-26 07:01:03.981735 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 960.101875 seconds old, received at 2015-10-26 06:45:03.879824: osd_op(client.23194930.0:4774508 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~94208] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 07:01:03.981756 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 960.101019 seconds old, received at 2015-10-26 06:45:03.880680: osd_op(client.23194930.0:4774509 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~131072] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 07:01:03.981758 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 960.100764 seconds old, received at 2015-10-26 06:45:03.880935: osd_op(client.23194930.0:4774510 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~524288] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 07:03:29.809648 7f552e0d2700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.247:6815/1021279 pipe(0xe2c56000 sd=153 :44360 s=2 pgs=2993 cs=7 l=0 c=0xf4aa35a0).fault with nothing to send, going to standby
|
|
2015-10-26 07:03:29.810739 7f552dce9700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.247:6813/27982 pipe(0x14a03b000 sd=425 :6811 s=2 pgs=2791 cs=7 l=0 c=0xeaa709a0).fault with nothing to send, going to standby
|
|
2015-10-26 07:05:36.387494 7f54eb0ae700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.251:6801/30678 pipe(0x14a4ad000 sd=155 :52457 s=2 pgs=2027 cs=2373 l=0 c=0x1dfe8260).fault with nothing to send, going to standby
|
|
2015-10-26 07:07:16.494616 7f54da998700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.240:6802/18909 pipe(0x25e88000 sd=379 :40317 s=2 pgs=124 cs=5 l=0 c=0x1016d34a0).fault with nothing to send, going to standby
|
|
2015-10-26 07:17:04.418357 7f55417cf700 0 log_channel(cluster) log [WRN] : 10 slow requests, 3 included below; oldest blocked for > 17984.567001 secs
|
|
2015-10-26 07:17:04.418363 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 1920.538500 seconds old, received at 2015-10-26 06:45:03.879824: osd_op(client.23194930.0:4774508 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~94208] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 07:17:04.418368 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 1920.537644 seconds old, received at 2015-10-26 06:45:03.880680: osd_op(client.23194930.0:4774509 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~131072] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 07:17:04.418371 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 1920.537389 seconds old, received at 2015-10-26 06:45:03.880935: osd_op(client.23194930.0:4774510 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~524288] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 07:20:23.098112 7f55417cf700 0 log_channel(cluster) log [WRN] : 10 slow requests, 3 included below; oldest blocked for > 18183.246747 secs
|
|
2015-10-26 07:20:23.098119 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 7680.254737 seconds old, received at 2015-10-26 05:12:22.843333: osd_op(client.23195434.0:3005325 rbd_data.446c0b74b0dc51.00000000000007ac [sparse-read 626176~4096] 58.61f9c444 ack+read+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 07:20:23.098124 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 7680.254489 seconds old, received at 2015-10-26 05:12:22.843581: osd_op(client.23195434.0:3005326 rbd_data.446c0b74b0dc51.00000000000007ac [sparse-read 634368~4096] 58.61f9c444 ack+read+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 07:20:23.098128 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 7680.254392 seconds old, received at 2015-10-26 05:12:22.843678: osd_op(client.23195434.0:3005327 rbd_data.446c0b74b0dc51.00000000000007ac [sparse-read 646656~4096] 58.61f9c444 ack+read+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 07:20:53.071426 7f54dec27700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.253:6803/2983 pipe(0xfd6fe000 sd=107 :44119 s=2 pgs=1774 cs=7 l=0 c=0xe809fde0).fault with nothing to send, going to standby
|
|
2015-10-26 07:23:58.108417 7f54da998700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.240:6802/18909 pipe(0x25e88000 sd=351 :50437 s=2 pgs=142 cs=7 l=0 c=0x1016d34a0).fault with nothing to send, going to standby
|
|
2015-10-26 07:26:05.400517 7f552d120700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.244:6801/11164 pipe(0xe6a6d000 sd=135 :58082 s=2 pgs=2929 cs=7 l=0 c=0xe60247e0).fault with nothing to send, going to standby
|
|
2015-10-26 07:30:04.486746 7f552d3e5700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.245:6805/2004640 pipe(0xd78ac000 sd=150 :6811 s=2 pgs=751 cs=3 l=0 c=0xd8748ec0).fault with nothing to send, going to standby
|
|
2015-10-26 07:31:09.491305 7f550e040700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.244:6806/18405 pipe(0xd895f000 sd=29 :58773 s=2 pgs=2495 cs=11 l=0 c=0xe6023e40).fault with nothing to send, going to standby
|
|
2015-10-26 07:33:52.372425 7f55331fb700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.239:6820/105498 pipe(0xdefc8000 sd=229 :38919 s=2 pgs=140 cs=9 l=0 c=0x14bbf6c60).fault with nothing to send, going to standby
|
|
2015-10-26 07:38:10.542596 7f54d2787700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.239:6801/111273 pipe(0xe5450000 sd=62 :6811 s=2 pgs=122 cs=5 l=0 c=0x10c6a3440).fault with nothing to send, going to standby
|
|
2015-10-26 07:40:11.114599 7f552e0d2700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.247:6815/1021279 pipe(0xe2c56000 sd=153 :51931 s=2 pgs=3007 cs=9 l=0 c=0xf4aa35a0).fault with nothing to send, going to standby
|
|
2015-10-26 07:45:03.747472 7f54f3734700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.245:6805/2004640 pipe(0x15045e000 sd=256 :6811 s=0 pgs=0 cs=0 l=0 c=0xe809dce0).accept connect_seq 4 vs existing 3 state standby
|
|
2015-10-26 07:46:48.091400 7f54dec27700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.253:6803/2983 pipe(0xfd6fe000 sd=107 :48803 s=2 pgs=1782 cs=9 l=0 c=0xe809fde0).fault with nothing to send, going to standby
|
|
2015-10-26 07:47:54.082558 7f54f0a07700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.253:6803/2983 pipe(0x19433a000 sd=435 :6811 s=0 pgs=0 cs=0 l=0 c=0x142e343c0).accept connect_seq 10 vs existing 9 state standby
|
|
2015-10-26 07:47:55.593546 7f553e892700 0 -- 192.168.99.251:6811/13454 >> 192.168.99.239:6817/105915 pipe(0x19433f000 sd=252 :6811 s=0 pgs=0 cs=0 l=0 c=0x142e33fa0).accept connect_seq 38 vs existing 37 state standby
|
|
2015-10-26 07:49:11.864639 7f54ff660700 0 log_channel(cluster) log [INF] : 58.195 restarting backfill on osd.15 from (89430'7647299,89446'7650309] MAX to 90281'7663038
|
|
2015-10-26 07:49:12.017413 7f55417cf700 0 log_channel(cluster) log [WRN] : 38 slow requests, 31 included below; oldest blocked for > 19912.165750 secs
|
|
2015-10-26 07:49:12.017418 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 70.568737 seconds old, received at 2015-10-26 07:48:01.448336: osd_op(client.23214218.0:2494238 rbd_header.f870572ae8944a [watch ping cookie 139931108247744] 58.2e4bea15 ondisk+write+known_if_redirected e91709) currently no flag points reached
|
|
2015-10-26 07:49:12.017431 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 43.237287 seconds old, received at 2015-10-26 07:48:28.779786: osd_repop_reply(osd.0.0:55485769 58.67 ondisk, result = 0) currently no flag points reached
|
|
2015-10-26 07:49:12.017433 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 70.447395 seconds old, received at 2015-10-26 07:48:01.569678: osd_op(client.23186943.0:2342383 rbd_header.41467d2ae8944a [watch ping cookie 139885272945904] 58.d44eae66 ondisk+write+known_if_redirected e91709) currently no flag points reached
|
|
2015-10-26 07:49:12.017436 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 51.529761 seconds old, received at 2015-10-26 07:48:20.487312: osd_repop_reply(osd.0.0:55485761 58.238 ondisk, result = 0) currently no flag points reached
|
|
2015-10-26 07:49:12.017437 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 42.977310 seconds old, received at 2015-10-26 07:48:29.039763: osd_repop_reply(osd.0.0:55485769 58.67 ondisk, result = 0) currently no flag points reached
|
|
2015-10-26 07:49:12.686016 7f54ff660700 0 log_channel(cluster) log [INF] : 58.168 restarting backfill on osd.13 from (0'0,0'0] MAX to 90157'6934421
|
|
2015-10-26 07:49:13.168847 7f55417cf700 0 log_channel(cluster) log [WRN] : 39 slow requests, 29 included below; oldest blocked for > 19913.317459 secs
|
|
2015-10-26 07:49:13.168854 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 71.720447 seconds old, received at 2015-10-26 07:48:01.448336: osd_op(client.23214218.0:2494238 rbd_header.f870572ae8944a [watch ping cookie 139931108247744] 58.2e4bea15 ondisk+write+known_if_redirected e91709) currently no flag points reached
|
|
2015-10-26 07:49:13.168858 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 71.599105 seconds old, received at 2015-10-26 07:48:01.569678: osd_op(client.23186943.0:2342383 rbd_header.41467d2ae8944a [watch ping cookie 139885272945904] 58.d44eae66 ondisk+write+known_if_redirected e91709) currently no flag points reached
|
|
2015-10-26 07:49:13.168862 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 52.640554 seconds old, received at 2015-10-26 07:48:20.528229: osd_repop_reply(osd.0.0:55485761 58.238 ondisk, result = 0) currently no flag points reached
|
|
2015-10-26 07:49:13.168865 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 42.837332 seconds old, received at 2015-10-26 07:48:30.331451: osd_repop(client.29173548.0:4979996 58.d5 613478d5/rbd_data.436fd12ae8944a.0000000000000535/head//58 v 91737'4832804) currently no flag points reached
|
|
2015-10-26 07:49:13.168869 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 70.477277 seconds old, received at 2015-10-26 07:48:02.691506: osd_op(client.23188857.0:2342134 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91710) currently no flag points reached
|
|
2015-10-26 07:49:23.101689 7f55417cf700 0 log_channel(cluster) log [WRN] : 39 slow requests, 28 included below; oldest blocked for > 19923.250302 secs
|
|
2015-10-26 07:49:23.101695 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 62.614313 seconds old, received at 2015-10-26 07:48:20.487312: osd_repop_reply(osd.0.0:55485761 58.238 ondisk, result = 0) currently no flag points reached
|
|
2015-10-26 07:49:23.101697 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 62.573397 seconds old, received at 2015-10-26 07:48:20.528229: osd_repop_reply(osd.0.0:55485761 58.238 ondisk, result = 0) currently no flag points reached
|
|
2015-10-26 07:49:23.101700 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 80.410119 seconds old, received at 2015-10-26 07:48:02.691506: osd_op(client.23188857.0:2342134 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91710) currently no flag points reached
|
|
2015-10-26 07:49:23.101703 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 80.300402 seconds old, received at 2015-10-26 07:48:02.801224: osd_op(client.23196607.0:43022676 volume-4d8a330f-69e2-4148-bba4-ab182925f93d.rbd [watch ping cookie 140717288596864] 58.38a8ca15 ondisk+write+known_if_redirected e91710) currently no flag points reached
|
|
2015-10-26 07:49:23.101705 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 50.191485 seconds old, received at 2015-10-26 07:48:32.910140: osd_repop(client.28525236.0:562125 58.176 cdb18b76/rbd_header.4415dc2ae8944a/head//58 v 91740'4002181) currently no flag points reached
|
|
2015-10-26 07:49:52.036812 7f55417cf700 0 log_channel(cluster) log [WRN] : 40 slow requests, 32 included below; oldest blocked for > 19952.185442 secs
|
|
2015-10-26 07:49:52.036844 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 83.256980 seconds old, received at 2015-10-26 07:48:28.779786: osd_repop_reply(osd.0.0:55485769 58.67 ondisk, result = 0) currently no flag points reached
|
|
2015-10-26 07:49:52.036848 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 38.867767 seconds old, received at 2015-10-26 07:49:13.168999: osd_op(client.23912404.0:2205173 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91723) currently no flag points reached
|
|
2015-10-26 07:49:52.036851 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 38.864447 seconds old, received at 2015-10-26 07:49:13.172318: osd_op(client.30354235.0:63111 rbd_header.1043be34948a717 [watch ping cookie 140163372123936] 58.75d6e46b ondisk+write+known_if_redirected e91733) currently no flag points reached
|
|
2015-10-26 07:49:52.036853 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 38.873849 seconds old, received at 2015-10-26 07:49:13.162916: osd_op(client.23188857.0:2342137 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91720) currently reached_pg
|
|
2015-10-26 07:49:52.036856 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 81.705314 seconds old, received at 2015-10-26 07:48:30.331451: osd_repop(client.29173548.0:4979996 58.d5 613478d5/rbd_data.436fd12ae8944a.0000000000000535/head//58 v 91737'4832804) currently no flag points reached
|
|
2015-10-26 07:50:14.285429 7f55417cf700 0 log_channel(cluster) log [WRN] : 36 slow requests, 29 included below; oldest blocked for > 19974.434064 secs
|
|
2015-10-26 07:50:14.285434 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 61.116389 seconds old, received at 2015-10-26 07:49:13.168999: osd_op(client.23912404.0:2205173 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91723) currently reached_pg
|
|
2015-10-26 07:50:14.285764 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 61.122471 seconds old, received at 2015-10-26 07:49:13.162916: osd_op(client.23188857.0:2342137 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91720) currently reached_pg
|
|
2015-10-26 07:50:14.285768 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 61.122493 seconds old, received at 2015-10-26 07:49:13.162894: osd_op(client.23210240.0:2422039 rbd_data.436a3d2ae8944a.00000000000009e3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3259904~4096] 58.edd7b6a6 ack+ondisk+write+known_if_redirected e91756) currently reached_pg
|
|
2015-10-26 07:50:14.285772 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 131.593881 seconds old, received at 2015-10-26 07:48:02.691506: osd_op(client.23188857.0:2342134 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91710) currently reached_pg
|
|
2015-10-26 07:50:14.285775 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 61.122451 seconds old, received at 2015-10-26 07:49:13.162937: osd_op(client.30354811.0:111966 rbd_data.7407492ae8944a.0000000000000418 [set-alloc-hint object_size 4194304 write_size 4194304,write 2871296~4096] 58.7fcc5662 ack+ondisk+write+known_if_redirected e91766) currently waiting for subops from 18,27
|
|
2015-10-26 07:50:42.298802 7f55417cf700 0 log_channel(cluster) log [WRN] : 37 slow requests, 27 included below; oldest blocked for > 20002.447443 secs
|
|
2015-10-26 07:50:42.298807 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 50.258630 seconds old, received at 2015-10-26 07:49:52.040136: osd_op(client.23211791.0:3604217 rbd_data.4177132ae8944a.0000000000000e40 [sparse-read 2011136~4096] 58.5435493e ack+read+known_if_redirected e91786) currently no flag points reached
|
|
2015-10-26 07:50:42.298812 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 89.135872 seconds old, received at 2015-10-26 07:49:13.162894: osd_op(client.23210240.0:2422039 rbd_data.436a3d2ae8944a.00000000000009e3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3259904~4096] 58.edd7b6a6 ack+ondisk+write+known_if_redirected e91756) currently reached_pg
|
|
2015-10-26 07:50:42.298816 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 89.135830 seconds old, received at 2015-10-26 07:49:13.162937: osd_op(client.30354811.0:111966 rbd_data.7407492ae8944a.0000000000000418 [set-alloc-hint object_size 4194304 write_size 4194304,write 2871296~4096] 58.7fcc5662 ack+ondisk+write+known_if_redirected e91766) currently waiting for subops from 18,27
|
|
2015-10-26 07:50:42.298820 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 89.134124 seconds old, received at 2015-10-26 07:49:13.164642: osd_op(client.31349106.0:63963 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91722) currently reached_pg
|
|
2015-10-26 07:50:42.298823 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 168.276995 seconds old, received at 2015-10-26 07:47:54.021772: osd_op(client.31349106.0:63957 rbd_header.41415b2ae8944a [watch reconnect cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:50:46.399794 7f55417cf700 0 log_channel(cluster) log [WRN] : 39 slow requests, 26 included below; oldest blocked for > 20006.548417 secs
|
|
2015-10-26 07:50:46.399802 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 93.235098 seconds old, received at 2015-10-26 07:49:13.164642: osd_op(client.31349106.0:63963 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91722) currently reached_pg
|
|
2015-10-26 07:50:46.399816 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 172.377968 seconds old, received at 2015-10-26 07:47:54.021772: osd_op(client.31349106.0:63957 rbd_header.41415b2ae8944a [watch reconnect cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:50:46.399820 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 93.224223 seconds old, received at 2015-10-26 07:49:13.175517: osd_op(client.23188857.0:2342140 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91735) currently reached_pg
|
|
2015-10-26 07:50:46.399824 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 172.374834 seconds old, received at 2015-10-26 07:47:54.024906: osd_op(client.30611758.0:13505 rbd_header.159b1232ae8944a [watch reconnect cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:50:46.399827 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 161.191889 seconds old, received at 2015-10-26 07:48:05.207852: osd_op(client.30611758.0:13508 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91712) currently reached_pg
|
|
2015-10-26 07:50:47.735003 7f55417cf700 0 log_channel(cluster) log [WRN] : 39 slow requests, 25 included below; oldest blocked for > 20007.883643 secs
|
|
2015-10-26 07:50:47.735010 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 173.713194 seconds old, received at 2015-10-26 07:47:54.021772: osd_op(client.31349106.0:63957 rbd_header.41415b2ae8944a [watch reconnect cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:50:47.735014 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 94.559449 seconds old, received at 2015-10-26 07:49:13.175517: osd_op(client.23188857.0:2342140 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91735) currently reached_pg
|
|
2015-10-26 07:50:47.735017 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 173.710060 seconds old, received at 2015-10-26 07:47:54.024906: osd_op(client.30611758.0:13505 rbd_header.159b1232ae8944a [watch reconnect cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:50:47.735020 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 162.527114 seconds old, received at 2015-10-26 07:48:05.207852: osd_op(client.30611758.0:13508 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91712) currently reached_pg
|
|
2015-10-26 07:50:47.735024 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 94.558710 seconds old, received at 2015-10-26 07:49:13.176256: osd_op(client.30611758.0:13514 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91737) currently reached_pg
|
|
2015-10-26 07:50:48.803474 7f54e5d57700 0 -- 192.168.99.251:6804/13454 >> 192.168.99.252:6789/0 pipe(0x2c5f4000 sd=291 :0 s=1 pgs=0 cs=0 l=1 c=0x155097c80).fault
|
|
2015-10-26 07:50:51.285974 7f55417cf700 0 log_channel(cluster) log [WRN] : 40 slow requests, 24 included below; oldest blocked for > 20011.434609 secs
|
|
2015-10-26 07:50:51.285980 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 177.261027 seconds old, received at 2015-10-26 07:47:54.024906: osd_op(client.30611758.0:13505 rbd_header.159b1232ae8944a [watch reconnect cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:50:51.285983 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 166.078081 seconds old, received at 2015-10-26 07:48:05.207852: osd_op(client.30611758.0:13508 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91712) currently reached_pg
|
|
2015-10-26 07:50:51.285985 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 98.109677 seconds old, received at 2015-10-26 07:49:13.176256: osd_op(client.30611758.0:13514 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91737) currently reached_pg
|
|
2015-10-26 07:50:51.285987 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 177.254476 seconds old, received at 2015-10-26 07:47:54.031456: osd_op(client.23188857.0:2342132 rbd_header.435cba2ae8944a [watch reconnect cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:50:51.285990 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 165.612201 seconds old, received at 2015-10-26 07:48:05.673731: osd_op(client.31349106.0:63960 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91713) currently reached_pg
|
|
2015-10-26 07:51:00.585479 7f55417cf700 0 log_channel(cluster) log [WRN] : 41 slow requests, 23 included below; oldest blocked for > 20020.734098 secs
|
|
2015-10-26 07:51:00.585487 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 68.545284 seconds old, received at 2015-10-26 07:49:52.040136: osd_op(client.23211791.0:3604217 rbd_data.4177132ae8944a.0000000000000e40 [sparse-read 2011136~4096] 58.5435493e ack+read+known_if_redirected e91786) currently reached_pg
|
|
2015-10-26 07:51:00.585492 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 186.553965 seconds old, received at 2015-10-26 07:47:54.031456: osd_op(client.23188857.0:2342132 rbd_header.435cba2ae8944a [watch reconnect cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:51:00.585659 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 174.911690 seconds old, received at 2015-10-26 07:48:05.673731: osd_op(client.31349106.0:63960 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91713) currently reached_pg
|
|
2015-10-26 07:51:00.585663 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 174.392858 seconds old, received at 2015-10-26 07:48:06.192562: osd_op(client.23912404.0:2205169 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91713) currently reached_pg
|
|
2015-10-26 07:51:00.585667 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 107.408949 seconds old, received at 2015-10-26 07:49:13.176472: osd_op(client.23912404.0:2205175 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91733) currently reached_pg
|
|
2015-10-26 07:51:01.587703 7f55417cf700 0 log_channel(cluster) log [WRN] : 41 slow requests, 22 included below; oldest blocked for > 20021.736321 secs
|
|
2015-10-26 07:51:01.587709 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 187.556188 seconds old, received at 2015-10-26 07:47:54.031456: osd_op(client.23188857.0:2342132 rbd_header.435cba2ae8944a [watch reconnect cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:51:01.587714 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 175.913913 seconds old, received at 2015-10-26 07:48:05.673731: osd_op(client.31349106.0:63960 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91713) currently reached_pg
|
|
2015-10-26 07:51:01.587718 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 175.395081 seconds old, received at 2015-10-26 07:48:06.192562: osd_op(client.23912404.0:2205169 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91713) currently reached_pg
|
|
2015-10-26 07:51:01.587722 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 108.411172 seconds old, received at 2015-10-26 07:49:13.176472: osd_op(client.23912404.0:2205175 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91733) currently reached_pg
|
|
2015-10-26 07:51:01.587726 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 132.742810 seconds old, received at 2015-10-26 07:48:48.844834: osd_op(client.23210240.0:2422037 rbd_data.436a3d2ae8944a.00000000000009c4 [set-alloc-hint object_size 4194304 write_size 4194304,write 126464~4096] 58.f701da35 ack+ondisk+write+known_if_redirected e91755) currently reached_pg
|
|
2015-10-26 07:51:03.168806 7f55417cf700 0 log_channel(cluster) log [WRN] : 42 slow requests, 20 included below; oldest blocked for > 20023.317416 secs
|
|
2015-10-26 07:51:03.168810 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 176.976177 seconds old, received at 2015-10-26 07:48:06.192562: osd_op(client.23912404.0:2205169 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91713) currently reached_pg
|
|
2015-10-26 07:51:03.168824 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 134.323905 seconds old, received at 2015-10-26 07:48:48.844834: osd_op(client.23210240.0:2422037 rbd_data.436a3d2ae8944a.00000000000009c4 [set-alloc-hint object_size 4194304 write_size 4194304,write 126464~4096] 58.f701da35 ack+ondisk+write+known_if_redirected e91755) currently reached_pg
|
|
2015-10-26 07:51:03.168826 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 109.997816 seconds old, received at 2015-10-26 07:49:13.170923: osd_op(client.23188857.0:2342138 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91725) currently reached_pg
|
|
2015-10-26 07:51:03.168828 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 110.009324 seconds old, received at 2015-10-26 07:49:13.159415: osd_op(client.23188857.0:2342136 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91717) currently reached_pg
|
|
2015-10-26 07:51:03.168831 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 175.477133 seconds old, received at 2015-10-26 07:48:07.691606: osd_op(client.23188857.0:2342135 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91714) currently reached_pg
|
|
2015-10-26 07:51:53.515573 7f54e22d5700 0 -- 192.168.99.251:6804/13454 >> 192.168.99.252:6789/0 pipe(0xd68f1000 sd=241 :0 s=1 pgs=0 cs=0 l=1 c=0x56e1de0).fault
|
|
2015-10-26 07:51:59.750711 7f55417cf700 0 log_channel(cluster) log [WRN] : 37 slow requests, 28 included below; oldest blocked for > 20079.899341 secs
|
|
2015-10-26 07:51:59.750717 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 166.581666 seconds old, received at 2015-10-26 07:49:13.168999: osd_op(client.23912404.0:2205173 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91723) currently reached_pg
|
|
2015-10-26 07:51:59.750721 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 166.587748 seconds old, received at 2015-10-26 07:49:13.162916: osd_op(client.23188857.0:2342137 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91720) currently reached_pg
|
|
2015-10-26 07:51:59.750724 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 127.710528 seconds old, received at 2015-10-26 07:49:52.040136: osd_op(client.23211791.0:3604217 rbd_data.4177132ae8944a.0000000000000e40 [sparse-read 2011136~4096] 58.5435493e ack+read+known_if_redirected e91786) currently reached_pg
|
|
2015-10-26 07:51:59.750727 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 166.587770 seconds old, received at 2015-10-26 07:49:13.162894: osd_op(client.23210240.0:2422039 rbd_data.436a3d2ae8944a.00000000000009e3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3259904~4096] 58.edd7b6a6 ack+ondisk+write+known_if_redirected e91756) currently reached_pg
|
|
2015-10-26 07:51:59.750730 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 166.587728 seconds old, received at 2015-10-26 07:49:13.162937: osd_op(client.30354811.0:111966 rbd_data.7407492ae8944a.0000000000000418 [set-alloc-hint object_size 4194304 write_size 4194304,write 2871296~4096] 58.7fcc5662 ack+ondisk+write+known_if_redirected e91766) currently waiting for subops from 18,27
|
|
2015-10-26 07:52:03.382860 7f54ec4c2700 0 -- 192.168.99.251:6804/13454 >> 192.168.99.251:0/2029859 pipe(0xd68f1000 sd=442 :6804 s=0 pgs=0 cs=0 l=1 c=0x10b710260).accept replacing existing (lossy) channel (new one lossy=1)
|
|
2015-10-26 07:52:11.152211 7f55417cf700 0 log_channel(cluster) log [WRN] : 38 slow requests, 26 included below; oldest blocked for > 20091.300835 secs
|
|
2015-10-26 07:52:11.152219 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 248.460653 seconds old, received at 2015-10-26 07:48:02.691506: osd_op(client.23188857.0:2342134 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91710) currently reached_pg
|
|
2015-10-26 07:52:11.152224 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 177.987516 seconds old, received at 2015-10-26 07:49:13.164642: osd_op(client.31349106.0:63963 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91722) currently reached_pg
|
|
2015-10-26 07:52:11.152227 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 257.130387 seconds old, received at 2015-10-26 07:47:54.021772: osd_op(client.31349106.0:63957 rbd_header.41415b2ae8944a [watch reconnect cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:11.152231 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 177.976642 seconds old, received at 2015-10-26 07:49:13.175517: osd_op(client.23188857.0:2342140 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91735) currently reached_pg
|
|
2015-10-26 07:52:11.152234 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 257.127253 seconds old, received at 2015-10-26 07:47:54.024906: osd_op(client.30611758.0:13505 rbd_header.159b1232ae8944a [watch reconnect cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:15.683962 7f54e39ec700 0 -- 192.168.99.251:6804/13454 >> 192.168.99.251:0/1018598 pipe(0xe8c54000 sd=455 :6804 s=0 pgs=0 cs=0 l=1 c=0x10b711c80).accept replacing existing (lossy) channel (new one lossy=1)
|
|
2015-10-26 07:52:17.097499 7f553e461700 0 -- 192.168.99.251:6804/13454 >> 192.168.99.239:0/3084481 pipe(0x16eba2000 sd=457 :6804 s=0 pgs=0 cs=0 l=1 c=0x10b70f8c0).accept replacing existing (lossy) channel (new one lossy=1)
|
|
2015-10-26 07:52:23.234595 7f55417cf700 0 log_channel(cluster) log [WRN] : 41 slow requests, 24 included below; oldest blocked for > 20103.383236 secs
|
|
2015-10-26 07:52:23.234599 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 258.026707 seconds old, received at 2015-10-26 07:48:05.207852: osd_op(client.30611758.0:13508 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91712) currently reached_pg
|
|
2015-10-26 07:52:23.234601 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 190.058303 seconds old, received at 2015-10-26 07:49:13.176256: osd_op(client.30611758.0:13514 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91737) currently reached_pg
|
|
2015-10-26 07:52:23.234604 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 269.203103 seconds old, received at 2015-10-26 07:47:54.031456: osd_op(client.23188857.0:2342132 rbd_header.435cba2ae8944a [watch reconnect cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:23.234605 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 257.560828 seconds old, received at 2015-10-26 07:48:05.673731: osd_op(client.31349106.0:63960 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91713) currently reached_pg
|
|
2015-10-26 07:52:23.234608 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 257.041996 seconds old, received at 2015-10-26 07:48:06.192562: osd_op(client.23912404.0:2205169 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91713) currently reached_pg
|
|
2015-10-26 07:52:26.770640 7f55417cf700 0 log_channel(cluster) log [WRN] : 45 slow requests, 23 included below; oldest blocked for > 20106.919250 secs
|
|
2015-10-26 07:52:26.770647 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 49.799651 seconds old, received at 2015-10-26 07:51:36.970923: osd_op(client.23195146.0:3007232 rbd_data.44355674b0dc51.0000000000000467 [set-alloc-hint object_size 4194304 write_size 4194304,write 4177920~4096] 58.6a297a7d ack+ondisk+write+known_if_redirected e91868) currently no flag points reached
|
|
2015-10-26 07:52:26.770651 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 43.514100 seconds old, received at 2015-10-26 07:51:43.256473: osd_op(client.28999844.0:11503725 rbd_data.1b1792840d07d5b.0000000000000079@2d [sparse-read 7557120~24576] 58.bb4dcf35 ack+read+localize_reads+known_if_redirected e91868) currently no flag points reached
|
|
2015-10-26 07:52:26.770655 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 193.594102 seconds old, received at 2015-10-26 07:49:13.176472: osd_op(client.23912404.0:2205175 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91733) currently reached_pg
|
|
2015-10-26 07:52:26.770659 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 150.755016 seconds old, received at 2015-10-26 07:49:56.015558: osd_op(client.30611758.0:13515 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91742) currently reached_pg
|
|
2015-10-26 07:52:26.770662 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 217.925740 seconds old, received at 2015-10-26 07:48:48.844834: osd_op(client.23210240.0:2422037 rbd_data.436a3d2ae8944a.00000000000009c4 [set-alloc-hint object_size 4194304 write_size 4194304,write 126464~4096] 58.f701da35 ack+ondisk+write+known_if_redirected e91755) currently reached_pg
|
|
2015-10-26 07:52:30.553002 7f55417cf700 0 log_channel(cluster) log [WRN] : 45 slow requests, 20 included below; oldest blocked for > 20110.701620 secs
|
|
2015-10-26 07:52:30.553006 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 154.537385 seconds old, received at 2015-10-26 07:49:56.015558: osd_op(client.30611758.0:13515 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91742) currently reached_pg
|
|
2015-10-26 07:52:30.553009 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 197.382020 seconds old, received at 2015-10-26 07:49:13.170923: osd_op(client.23188857.0:2342138 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91725) currently reached_pg
|
|
2015-10-26 07:52:30.553012 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 197.393527 seconds old, received at 2015-10-26 07:49:13.159415: osd_op(client.23188857.0:2342136 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91717) currently reached_pg
|
|
2015-10-26 07:52:30.553210 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 40.287979 seconds old, received at 2015-10-26 07:51:50.264964: osd_op(client.30611677.0:40161 rbd_data.dd81e52ae8944a.00000000000052f0 [sparse-read 32256~4096] 58.51f4d68 ack+read+known_if_redirected e91871) currently reached_pg
|
|
2015-10-26 07:52:30.553215 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 262.861337 seconds old, received at 2015-10-26 07:48:07.691606: osd_op(client.23188857.0:2342135 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91714) currently reached_pg
|
|
2015-10-26 07:52:31.553368 7f55417cf700 0 log_channel(cluster) log [WRN] : 46 slow requests, 20 included below; oldest blocked for > 20111.701990 secs
|
|
2015-10-26 07:52:31.553374 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 155.537755 seconds old, received at 2015-10-26 07:49:56.015558: osd_op(client.30611758.0:13515 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91742) currently reached_pg
|
|
2015-10-26 07:52:31.553378 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 198.382390 seconds old, received at 2015-10-26 07:49:13.170923: osd_op(client.23188857.0:2342138 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91725) currently reached_pg
|
|
2015-10-26 07:52:31.553382 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 198.393898 seconds old, received at 2015-10-26 07:49:13.159415: osd_op(client.23188857.0:2342136 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91717) currently reached_pg
|
|
2015-10-26 07:52:31.553386 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 263.861707 seconds old, received at 2015-10-26 07:48:07.691606: osd_op(client.23188857.0:2342135 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91714) currently reached_pg
|
|
2015-10-26 07:52:31.553389 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 211.840859 seconds old, received at 2015-10-26 07:48:59.712454: osd_op(client.30354811.0:111963 rbd_data.7407492ae8944a.00000000000001fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3194880~28672] 58.60947401 ack+ondisk+write+known_if_redirected e91766) currently waiting for subops from 18,30
|
|
2015-10-26 07:52:32.651173 7f55417cf700 0 log_channel(cluster) log [WRN] : 48 slow requests, 19 included below; oldest blocked for > 20112.799806 secs
|
|
2015-10-26 07:52:32.651178 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 264.959523 seconds old, received at 2015-10-26 07:48:07.691606: osd_op(client.23188857.0:2342135 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91714) currently reached_pg
|
|
2015-10-26 07:52:32.651181 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 212.938676 seconds old, received at 2015-10-26 07:48:59.712454: osd_op(client.30354811.0:111963 rbd_data.7407492ae8944a.00000000000001fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3194880~28672] 58.60947401 ack+ondisk+write+known_if_redirected e91766) currently waiting for subops from 18,30
|
|
2015-10-26 07:52:32.651184 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 199.479485 seconds old, received at 2015-10-26 07:49:13.171645: osd_op(client.23188857.0:2342139 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91730) currently reached_pg
|
|
2015-10-26 07:52:32.651186 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 277.443629 seconds old, received at 2015-10-26 07:47:55.207500: osd_op(client.30611758.0:13506 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:32.651188 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 199.484396 seconds old, received at 2015-10-26 07:49:13.166733: osd_op(client.31349106.0:63964 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91728) currently reached_pg
|
|
2015-10-26 07:52:33.342588 7f54ff660700 0 log_channel(cluster) log [INF] : 58.14b restarting backfill on osd.15 from (0'0,0'0] MAX to 90122'6500802
|
|
2015-10-26 07:52:34.385616 7f55417cf700 0 log_channel(cluster) log [WRN] : 48 slow requests, 18 included below; oldest blocked for > 20114.534246 secs
|
|
2015-10-26 07:52:34.385619 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 214.673115 seconds old, received at 2015-10-26 07:48:59.712454: osd_op(client.30354811.0:111963 rbd_data.7407492ae8944a.00000000000001fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3194880~28672] 58.60947401 ack+ondisk+write+known_if_redirected e91766) currently waiting for subops from 18,30
|
|
2015-10-26 07:52:34.385622 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 201.213924 seconds old, received at 2015-10-26 07:49:13.171645: osd_op(client.23188857.0:2342139 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91730) currently reached_pg
|
|
2015-10-26 07:52:34.385624 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 279.178069 seconds old, received at 2015-10-26 07:47:55.207500: osd_op(client.30611758.0:13506 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:34.385627 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 201.218836 seconds old, received at 2015-10-26 07:49:13.166733: osd_op(client.31349106.0:63964 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91728) currently reached_pg
|
|
2015-10-26 07:52:34.385629 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 201.213877 seconds old, received at 2015-10-26 07:49:13.171692: osd_op(client.30611758.0:13513 rbd_header.159b1232ae8944a [watch ping cookie 140272088385728 gen 1] 58.ced3f638 ondisk+write+known_if_redirected e91732) currently reached_pg
|
|
2015-10-26 07:52:35.532641 7f55417cf700 0 log_channel(cluster) log [WRN] : 40 slow requests, 16 included below; oldest blocked for > 20115.681278 secs
|
|
2015-10-26 07:52:35.532646 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 202.360956 seconds old, received at 2015-10-26 07:49:13.171645: osd_op(client.23188857.0:2342139 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91730) currently reached_pg
|
|
2015-10-26 07:52:35.532650 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 202.365868 seconds old, received at 2015-10-26 07:49:13.166733: osd_op(client.31349106.0:63964 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91728) currently reached_pg
|
|
2015-10-26 07:52:35.532654 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 279.859136 seconds old, received at 2015-10-26 07:47:55.673466: osd_op(client.31349106.0:63958 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:35.532657 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 202.372739 seconds old, received at 2015-10-26 07:49:13.159862: osd_op(client.31349106.0:63962 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91719) currently reached_pg
|
|
2015-10-26 07:52:35.532660 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 202.360729 seconds old, received at 2015-10-26 07:49:13.171872: osd_op(client.31349106.0:63965 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91732) currently reached_pg
|
|
2015-10-26 07:52:36.533680 7f55417cf700 0 log_channel(cluster) log [WRN] : 41 slow requests, 15 included below; oldest blocked for > 20116.682318 secs
|
|
2015-10-26 07:52:36.533685 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 280.860176 seconds old, received at 2015-10-26 07:47:55.673466: osd_op(client.31349106.0:63958 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:36.533688 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 203.373779 seconds old, received at 2015-10-26 07:49:13.159862: osd_op(client.31349106.0:63962 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91719) currently reached_pg
|
|
2015-10-26 07:52:36.533690 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 203.361769 seconds old, received at 2015-10-26 07:49:13.171872: osd_op(client.31349106.0:63965 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91732) currently reached_pg
|
|
2015-10-26 07:52:36.533692 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 203.373727 seconds old, received at 2015-10-26 07:49:13.159914: osd_op(client.23912404.0:2205172 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91719) currently reached_pg
|
|
2015-10-26 07:52:36.533694 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 203.361758 seconds old, received at 2015-10-26 07:49:13.171883: osd_op(client.23912404.0:2205174 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91731) currently reached_pg
|
|
2015-10-26 07:52:38.639570 7f55417cf700 0 log_channel(cluster) log [WRN] : 41 slow requests, 16 included below; oldest blocked for > 20118.788208 secs
|
|
2015-10-26 07:52:38.639575 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 61.668608 seconds old, received at 2015-10-26 07:51:36.970923: osd_op(client.23195146.0:3007232 rbd_data.44355674b0dc51.0000000000000467 [set-alloc-hint object_size 4194304 write_size 4194304,write 4177920~4096] 58.6a297a7d ack+ondisk+write+known_if_redirected e91868) currently waiting for subops from 23,38
|
|
2015-10-26 07:52:38.639578 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 282.966065 seconds old, received at 2015-10-26 07:47:55.673466: osd_op(client.31349106.0:63958 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:38.639580 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 205.479668 seconds old, received at 2015-10-26 07:49:13.159862: osd_op(client.31349106.0:63962 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91719) currently reached_pg
|
|
2015-10-26 07:52:38.639582 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 205.467659 seconds old, received at 2015-10-26 07:49:13.171872: osd_op(client.31349106.0:63965 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91732) currently reached_pg
|
|
2015-10-26 07:52:38.639585 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 205.479617 seconds old, received at 2015-10-26 07:49:13.159914: osd_op(client.23912404.0:2205172 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91719) currently reached_pg
|
|
2015-10-26 07:52:39.672876 7f55417cf700 0 log_channel(cluster) log [WRN] : 42 slow requests, 14 included below; oldest blocked for > 20119.821510 secs
|
|
2015-10-26 07:52:39.672883 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 283.999368 seconds old, received at 2015-10-26 07:47:55.673466: osd_op(client.31349106.0:63958 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:39.672888 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 206.512919 seconds old, received at 2015-10-26 07:49:13.159914: osd_op(client.23912404.0:2205172 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91719) currently reached_pg
|
|
2015-10-26 07:52:39.672892 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 206.500950 seconds old, received at 2015-10-26 07:49:13.171883: osd_op(client.23912404.0:2205174 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91731) currently reached_pg
|
|
2015-10-26 07:52:39.672896 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 283.478958 seconds old, received at 2015-10-26 07:47:56.193876: osd_op(client.23912404.0:2205167 rbd_header.16cdf892ae8944a [watch reconnect cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91709) currently reached_pg
|
|
2015-10-26 07:52:39.672899 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 206.512809 seconds old, received at 2015-10-26 07:49:13.160024: osd_op(client.29183509.0:3545779 rbd_data.414aa92ae8944a.00000000000056a2 [sparse-read 302592~4096] 58.9c107f4b ack+read+known_if_redirected e91732) currently reached_pg
|
|
2015-10-26 07:52:40.673886 7f55417cf700 0 log_channel(cluster) log [WRN] : 41 slow requests, 12 included below; oldest blocked for > 20120.822519 secs
|
|
2015-10-26 07:52:40.673891 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 207.501959 seconds old, received at 2015-10-26 07:49:13.171883: osd_op(client.23912404.0:2205174 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91731) currently reached_pg
|
|
2015-10-26 07:52:40.673894 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 284.479967 seconds old, received at 2015-10-26 07:47:56.193876: osd_op(client.23912404.0:2205167 rbd_header.16cdf892ae8944a [watch reconnect cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91709) currently reached_pg
|
|
2015-10-26 07:52:40.673897 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 207.513819 seconds old, received at 2015-10-26 07:49:13.160024: osd_op(client.29183509.0:3545779 rbd_data.414aa92ae8944a.00000000000056a2 [sparse-read 302592~4096] 58.9c107f4b ack+read+known_if_redirected e91732) currently reached_pg
|
|
2015-10-26 07:52:40.673899 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 47.843092 seconds old, received at 2015-10-26 07:51:52.830751: osd_op(client.31033401.0:1411216 rbd_data.11f42e2ae8944a.000000000000031b [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~16384] 58.64ebaadd ack+ondisk+write+known_if_redirected e91874) currently waiting for subops from 9,39
|
|
2015-10-26 07:52:40.673901 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 269.999813 seconds old, received at 2015-10-26 07:48:10.674029: osd_op(client.31349106.0:63961 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91716) currently reached_pg
|
|
2015-10-26 07:52:41.181560 7f54ff660700 0 log_channel(cluster) log [INF] : 81.95b restarting backfill on osd.13 from (0'0,0'0] MAX to 89439'22322
|
|
2015-10-26 07:52:41.683026 7f55417cf700 0 log_channel(cluster) log [WRN] : 42 slow requests, 11 included below; oldest blocked for > 20121.831646 secs
|
|
2015-10-26 07:52:41.683033 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 285.489093 seconds old, received at 2015-10-26 07:47:56.193876: osd_op(client.23912404.0:2205167 rbd_header.16cdf892ae8944a [watch reconnect cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91709) currently reached_pg
|
|
2015-10-26 07:52:41.683048 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 208.522945 seconds old, received at 2015-10-26 07:49:13.160024: osd_op(client.29183509.0:3545779 rbd_data.414aa92ae8944a.00000000000056a2 [sparse-read 302592~4096] 58.9c107f4b ack+read+known_if_redirected e91732) currently reached_pg
|
|
2015-10-26 07:52:41.683052 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 271.008940 seconds old, received at 2015-10-26 07:48:10.674029: osd_op(client.31349106.0:63961 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91716) currently reached_pg
|
|
2015-10-26 07:52:41.683056 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 283.991497 seconds old, received at 2015-10-26 07:47:57.691472: osd_op(client.23188857.0:2342133 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:41.683060 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 270.490488 seconds old, received at 2015-10-26 07:48:11.192481: osd_op(client.23912404.0:2205171 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91716) currently reached_pg
|
|
2015-10-26 07:52:42.703839 7f55417cf700 0 log_channel(cluster) log [WRN] : 43 slow requests, 11 included below; oldest blocked for > 20122.852460 secs
|
|
2015-10-26 07:52:42.703847 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 286.509908 seconds old, received at 2015-10-26 07:47:56.193876: osd_op(client.23912404.0:2205167 rbd_header.16cdf892ae8944a [watch reconnect cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91709) currently reached_pg
|
|
2015-10-26 07:52:42.703851 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 272.029754 seconds old, received at 2015-10-26 07:48:10.674029: osd_op(client.31349106.0:63961 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91716) currently reached_pg
|
|
2015-10-26 07:52:42.703855 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 285.012311 seconds old, received at 2015-10-26 07:47:57.691472: osd_op(client.23188857.0:2342133 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:42.703859 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 271.511302 seconds old, received at 2015-10-26 07:48:11.192481: osd_op(client.23912404.0:2205171 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91716) currently reached_pg
|
|
2015-10-26 07:52:42.703876 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 4058.823959 seconds old, received at 2015-10-26 06:45:03.879824: osd_op(client.23194930.0:4774508 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~94208] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 07:52:43.939870 7f55417cf700 0 log_channel(cluster) log [WRN] : 44 slow requests, 10 included below; oldest blocked for > 20124.088486 secs
|
|
2015-10-26 07:52:43.939886 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 273.265780 seconds old, received at 2015-10-26 07:48:10.674029: osd_op(client.31349106.0:63961 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91716) currently reached_pg
|
|
2015-10-26 07:52:43.939890 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 286.248337 seconds old, received at 2015-10-26 07:47:57.691472: osd_op(client.23188857.0:2342133 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:43.939893 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 272.747328 seconds old, received at 2015-10-26 07:48:11.192481: osd_op(client.23912404.0:2205171 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91716) currently reached_pg
|
|
2015-10-26 07:52:43.939896 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 4060.059130 seconds old, received at 2015-10-26 06:45:03.880680: osd_op(client.23194930.0:4774509 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~131072] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 07:52:43.939900 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 32.897447 seconds old, received at 2015-10-26 07:52:11.042363: osd_op(client.23189310.0:4268208 rbd_data.622b633d1b58ba.0000000000016c25 [set-alloc-hint object_size 4194304 write_size 4194304,write 3317760~4096] 58.ebf6e7fc ack+ondisk+write+known_if_redirected e91881) currently waiting for subops from 9,33
|
|
2015-10-26 07:52:45.130113 7f55417cf700 0 log_channel(cluster) log [WRN] : 46 slow requests, 9 included below; oldest blocked for > 20125.278735 secs
|
|
2015-10-26 07:52:45.130121 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 287.438586 seconds old, received at 2015-10-26 07:47:57.691472: osd_op(client.23188857.0:2342133 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91707) currently reached_pg
|
|
2015-10-26 07:52:45.130126 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 273.937577 seconds old, received at 2015-10-26 07:48:11.192481: osd_op(client.23912404.0:2205171 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91716) currently reached_pg
|
|
2015-10-26 07:52:45.130130 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 4061.249123 seconds old, received at 2015-10-26 06:45:03.880935: osd_op(client.23194930.0:4774510 rbd_data.436cdd2ae8944a.0000000000000166 [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~524288] 58.f465444 ack+ondisk+write+known_if_redirected e91706) currently reached_pg
|
|
2015-10-26 07:52:45.130133 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 211.957850 seconds old, received at 2015-10-26 07:49:13.172208: osd_op(client.31349106.0:63966 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91737) currently reached_pg
|
|
2015-10-26 07:52:45.130137 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 139.193572 seconds old, received at 2015-10-26 07:50:25.936486: osd_op(client.23209583.0:4018440 rbd_data.419e2e2ae8944a.0000000000000900 [sparse-read 2399744~4096] 58.2b966d4b ack+read+known_if_redirected e91841) currently reached_pg
|
|
2015-10-26 07:52:46.324637 7f55417cf700 0 log_channel(cluster) log [WRN] : 47 slow requests, 8 included below; oldest blocked for > 20126.473258 secs
|
|
2015-10-26 07:52:46.324644 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 213.152373 seconds old, received at 2015-10-26 07:49:13.172208: osd_op(client.31349106.0:63966 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91737) currently reached_pg
|
|
2015-10-26 07:52:46.324658 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 140.388095 seconds old, received at 2015-10-26 07:50:25.936486: osd_op(client.23209583.0:4018440 rbd_data.419e2e2ae8944a.0000000000000900 [sparse-read 2399744~4096] 58.2b966d4b ack+read+known_if_redirected e91841) currently reached_pg
|
|
2015-10-26 07:52:46.324662 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 34.299135 seconds old, received at 2015-10-26 07:52:12.025446: osd_op(client.31047363.0:372270 rbd_data.17b84262ae8944a.0000000000003836 [set-alloc-hint object_size 4194304 write_size 4194304,write 3670016~4096] 58.27c45801 ack+ondisk+write+known_if_redirected e91882) currently waiting for subops from 18,30
|
|
2015-10-26 07:52:46.324665 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 285.651048 seconds old, received at 2015-10-26 07:48:00.673533: osd_op(client.31349106.0:63959 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91708) currently reached_pg
|
|
2015-10-26 07:52:46.324669 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 194.828134 seconds old, received at 2015-10-26 07:49:31.496447: osd_op(client.29173548.0:4980063 rbd_data.436fd12ae8944a.00000000000001c0 [sparse-read 249344~4096] 58.d07a5e0b ack+read+known_if_redirected e91798) currently reached_pg
|
|
2015-10-26 07:52:47.391608 7f55417cf700 0 log_channel(cluster) log [WRN] : 47 slow requests, 7 included below; oldest blocked for > 20127.540227 secs
|
|
2015-10-26 07:52:47.391613 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 214.219343 seconds old, received at 2015-10-26 07:49:13.172208: osd_op(client.31349106.0:63966 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91737) currently reached_pg
|
|
2015-10-26 07:52:47.391618 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 141.455064 seconds old, received at 2015-10-26 07:50:25.936486: osd_op(client.23209583.0:4018440 rbd_data.419e2e2ae8944a.0000000000000900 [sparse-read 2399744~4096] 58.2b966d4b ack+read+known_if_redirected e91841) currently reached_pg
|
|
2015-10-26 07:52:47.391622 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 286.718018 seconds old, received at 2015-10-26 07:48:00.673533: osd_op(client.31349106.0:63959 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91708) currently reached_pg
|
|
2015-10-26 07:52:47.391626 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 195.895104 seconds old, received at 2015-10-26 07:49:31.496447: osd_op(client.29173548.0:4980063 rbd_data.436fd12ae8944a.00000000000001c0 [sparse-read 249344~4096] 58.d07a5e0b ack+read+known_if_redirected e91798) currently reached_pg
|
|
2015-10-26 07:52:47.391629 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 184.243013 seconds old, received at 2015-10-26 07:49:43.148538: osd_op(client.31349106.0:63967 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91743) currently reached_pg
|
|
2015-10-26 07:52:48.754345 7f55417cf700 0 log_channel(cluster) log [WRN] : 47 slow requests, 6 included below; oldest blocked for > 20128.902973 secs
|
|
2015-10-26 07:52:48.754351 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 288.080764 seconds old, received at 2015-10-26 07:48:00.673533: osd_op(client.31349106.0:63959 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91708) currently reached_pg
|
|
2015-10-26 07:52:48.754355 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 197.257849 seconds old, received at 2015-10-26 07:49:31.496447: osd_op(client.29173548.0:4980063 rbd_data.436fd12ae8944a.00000000000001c0 [sparse-read 249344~4096] 58.d07a5e0b ack+read+known_if_redirected e91798) currently reached_pg
|
|
2015-10-26 07:52:48.754359 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 185.605759 seconds old, received at 2015-10-26 07:49:43.148538: osd_op(client.31349106.0:63967 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91743) currently reached_pg
|
|
2015-10-26 07:52:48.754361 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 287.561856 seconds old, received at 2015-10-26 07:48:01.192440: osd_op(client.23912404.0:2205168 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91709) currently reached_pg
|
|
2015-10-26 07:52:48.754364 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 259.870239 seconds old, received at 2015-10-26 07:48:28.884057: osd_op(client.29183509.0:3545778 rbd_data.414aa92ae8944a.00000000000056a2 [sparse-read 294400~4096] 58.9c107f4b ack+read+known_if_redirected e91732) currently reached_pg
|
|
2015-10-26 07:52:50.215498 7f55417cf700 0 log_channel(cluster) log [WRN] : 46 slow requests, 5 included below; oldest blocked for > 20130.364137 secs
|
|
2015-10-26 07:52:50.215503 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 289.541928 seconds old, received at 2015-10-26 07:48:00.673533: osd_op(client.31349106.0:63959 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91708) currently reached_pg
|
|
2015-10-26 07:52:50.215506 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 187.066923 seconds old, received at 2015-10-26 07:49:43.148538: osd_op(client.31349106.0:63967 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91743) currently reached_pg
|
|
2015-10-26 07:52:50.215508 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 289.023021 seconds old, received at 2015-10-26 07:48:01.192440: osd_op(client.23912404.0:2205168 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91709) currently reached_pg
|
|
2015-10-26 07:52:50.215510 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 261.331403 seconds old, received at 2015-10-26 07:48:28.884057: osd_op(client.29183509.0:3545778 rbd_data.414aa92ae8944a.00000000000056a2 [sparse-read 294400~4096] 58.9c107f4b ack+read+known_if_redirected e91732) currently reached_pg
|
|
2015-10-26 07:52:50.215512 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 187.000292 seconds old, received at 2015-10-26 07:49:43.215169: osd_op(client.23912404.0:2205176 rbd_header.16cdf892ae8944a [watch unwatch cookie 140387649913536] 58.6ea0fda5 ondisk+write+known_if_redirected e91738) currently reached_pg
|
|
2015-10-26 07:52:52.034435 7f55417cf700 0 log_channel(cluster) log [WRN] : 46 slow requests, 4 included below; oldest blocked for > 20132.183077 secs
|
|
2015-10-26 07:52:52.034440 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 61.769437 seconds old, received at 2015-10-26 07:51:50.264964: osd_op(client.30611677.0:40161 rbd_data.dd81e52ae8944a.00000000000052f0 [sparse-read 32256~4096] 58.51f4d68 ack+read+known_if_redirected e91871) currently reached_pg
|
|
2015-10-26 07:52:52.034444 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 290.841961 seconds old, received at 2015-10-26 07:48:01.192440: osd_op(client.23912404.0:2205168 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91709) currently reached_pg
|
|
2015-10-26 07:52:52.034446 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 263.150343 seconds old, received at 2015-10-26 07:48:28.884057: osd_op(client.29183509.0:3545778 rbd_data.414aa92ae8944a.00000000000056a2 [sparse-read 294400~4096] 58.9c107f4b ack+read+known_if_redirected e91732) currently reached_pg
|
|
2015-10-26 07:52:52.034448 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 188.819232 seconds old, received at 2015-10-26 07:49:43.215169: osd_op(client.23912404.0:2205176 rbd_header.16cdf892ae8944a [watch unwatch cookie 140387649913536] 58.6ea0fda5 ondisk+write+known_if_redirected e91738) currently reached_pg
|
|
2015-10-26 07:52:53.750878 7f55417cf700 0 log_channel(cluster) log [WRN] : 51 slow requests, 8 included below; oldest blocked for > 20133.899507 secs
|
|
2015-10-26 07:52:53.750884 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.507785 seconds old, received at 2015-10-26 07:52:23.243045: osd_op(client.31047363.0:372553 rbd_data.17b84262ae8944a.0000000000003836 [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~282624] 58.27c45801 ack+ondisk+write+known_if_redirected e91883) currently waiting for subops from 18,30
|
|
2015-10-26 07:52:53.750888 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.513790 seconds old, received at 2015-10-26 07:52:23.237040: osd_op(client.23188857.0:2342141 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91740) currently reached_pg
|
|
2015-10-26 07:52:53.750891 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.505629 seconds old, received at 2015-10-26 07:52:23.245201: osd_op(client.31349106.0:63968 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91747) currently reached_pg
|
|
2015-10-26 07:52:53.750895 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.508655 seconds old, received at 2015-10-26 07:52:23.242175: osd_op(client.30611677.0:40280 rbd_data.dd81e52ae8944a.0000000000004ae0 [sparse-read 2285056~8192] 58.1c6c2568 ack+read+known_if_redirected e91876) currently reached_pg
|
|
2015-10-26 07:52:53.750898 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.507718 seconds old, received at 2015-10-26 07:52:23.243112: osd_op(client.28525233.0:1290300 rbd_data.41609374b0dc51.00000000000004c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 2137600~4096] 58.b50d831c ack+ondisk+write+known_if_redirected e91884) currently reached_pg
|
|
2015-10-26 07:52:54.860039 7f55417cf700 0 log_channel(cluster) log [WRN] : 52 slow requests, 4 included below; oldest blocked for > 20135.008670 secs
|
|
2015-10-26 07:52:54.860044 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 31.616746 seconds old, received at 2015-10-26 07:52:23.243247: osd_op(client.23188857.0:2342142 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91745) currently reached_pg
|
|
2015-10-26 07:52:54.860047 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 293.667554 seconds old, received at 2015-10-26 07:48:01.192440: osd_op(client.23912404.0:2205168 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91709) currently reached_pg
|
|
2015-10-26 07:52:54.860051 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 265.975936 seconds old, received at 2015-10-26 07:48:28.884057: osd_op(client.29183509.0:3545778 rbd_data.414aa92ae8944a.00000000000056a2 [sparse-read 294400~4096] 58.9c107f4b ack+read+known_if_redirected e91732) currently reached_pg
|
|
2015-10-26 07:52:54.860054 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 191.644825 seconds old, received at 2015-10-26 07:49:43.215169: osd_op(client.23912404.0:2205176 rbd_header.16cdf892ae8944a [watch unwatch cookie 140387649913536] 58.6ea0fda5 ondisk+write+known_if_redirected e91738) currently reached_pg
|
|
2015-10-26 07:52:56.145310 7f55417cf700 0 log_channel(cluster) log [WRN] : 54 slow requests, 2 included below; oldest blocked for > 20136.293941 secs
|
|
2015-10-26 07:52:56.145316 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.549028 seconds old, received at 2015-10-26 07:52:25.596236: osd_op(client.29173392.0:807046 rbd_data.ca5da52ae8944a.00000000000000a6 [sparse-read 2732032~16384] 58.58fa6d3e ack+read+known_if_redirected e91886) currently reached_pg
|
|
2015-10-26 07:52:56.145320 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.875549 seconds old, received at 2015-10-26 07:52:25.269715: osd_op(client.30611677.0:40310 rbd_data.dd81e52ae8944a.0000000000005250 [sparse-read 48640~4096] 58.f0d3a953 ack+read+known_if_redirected e91877) currently reached_pg
|
|
2015-10-26 07:52:59.239891 7f55417cf700 0 log_channel(cluster) log [WRN] : 53 slow requests, 1 included below; oldest blocked for > 20139.388534 secs
|
|
2015-10-26 07:52:59.239898 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.348298 seconds old, received at 2015-10-26 07:52:28.891560: osd_op(client.23211368.0:41927684 rbd_data.447e1474b0dc51.0000000000004327 [sparse-read 2760192~16384] 58.e9ffea0b ack+read+known_if_redirected e91887) currently reached_pg
|
|
2015-10-26 07:53:00.424895 7f55417cf700 0 log_channel(cluster) log [WRN] : 54 slow requests, 1 included below; oldest blocked for > 20140.573532 secs
|
|
2015-10-26 07:53:00.424900 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.175993 seconds old, received at 2015-10-26 07:52:30.248862: osd_op(client.29183509.0:3545823 rbd_data.414aa92ae8944a.00000000000056a2 [sparse-read 0~40448] 58.9c107f4b ack+read+known_if_redirected e91794) currently reached_pg
|
|
2015-10-26 07:53:01.427434 7f54ff660700 0 log_channel(cluster) log [INF] : 81.334 restarting backfill on osd.13 from (0'0,0'0] MAX to 90122'34806
|
|
2015-10-26 07:53:01.429986 7f54ffe61700 0 log_channel(cluster) log [INF] : 58.11c restarting backfill on osd.13 from (0'0,0'0] MAX to 91706'3972239
|
|
2015-10-26 07:53:01.459063 7f54e5d57700 0 -- 192.168.99.251:6804/13454 >> 192.168.99.204:0/1100755602 pipe(0x178f1d000 sd=367 :6804 s=0 pgs=0 cs=0 l=0 c=0xd6faa7e0).accept peer addr is really 192.168.99.204:0/1100755602 (socket is 192.168.99.204:39961/0)
|
|
2015-10-26 07:53:02.004784 7f54ff660700 0 log_channel(cluster) log [INF] : 81.877 restarting backfill on osd.14 from (0'0,0'0] MAX to 90062'27108
|
|
2015-10-26 07:53:02.029896 7f54ffe61700 0 log_channel(cluster) log [INF] : 81.2a3 restarting backfill on osd.13 from (0'0,0'0] MAX to 90122'26155
|
|
2015-10-26 07:53:02.799501 7f54ffe61700 0 log_channel(cluster) log [INF] : 81.5e0 restarting backfill on osd.14 from (0'0,0'0] MAX to 89441'22508
|
|
2015-10-26 07:53:02.846348 7f54ff660700 0 log_channel(cluster) log [INF] : 81.2ca restarting backfill on osd.14 from (0'0,0'0] MAX to 89439'23661
|
|
2015-10-26 07:53:02.915407 7f55417cf700 0 log_channel(cluster) log [WRN] : 63 slow requests, 9 included below; oldest blocked for > 20143.064032 secs
|
|
2015-10-26 07:53:02.915412 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 31.798693 seconds old, received at 2015-10-26 07:52:31.116662: osd_op(client.31047363.0:373271 rbd_data.17b84262ae8944a.0000000000003673 [sparse-read 1753088~102400] 58.c7ea854b ack+read+known_if_redirected e91884) currently reached_pg
|
|
2015-10-26 07:53:02.915416 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 32.080110 seconds old, received at 2015-10-26 07:52:30.835246: osd_op(client.23211368.0:41927685 rbd_data.447e1474b0dc51.0000000000004327 [sparse-read 2780672~12288] 58.e9ffea0b ack+read+known_if_redirected e91887) currently reached_pg
|
|
2015-10-26 07:53:02.915420 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 31.218205 seconds old, received at 2015-10-26 07:52:31.697151: osd_op(client.25426561.0:20075469 rbd_data.d983732ae8944a.0000000000000047 [sparse-read 4104192~4096] 58.5bcabb1c ack+read+known_if_redirected e91890) currently reached_pg
|
|
2015-10-26 07:53:02.915423 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 31.083761 seconds old, received at 2015-10-26 07:52:31.831594: osd_op(client.31047363.0:377373 rbd_data.17b84262ae8944a.0000000000003673 [sparse-read 1609728~131072] 58.c7ea854b ack+read+known_if_redirected e91884) currently reached_pg
|
|
2015-10-26 07:53:02.915426 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 31.163243 seconds old, received at 2015-10-26 07:52:31.752113: osd_op(client.23189310.0:4268273 rbd_data.622b633d1b58ba.0000000000016a00 [set-alloc-hint object_size 4194304 write_size 4194304,write 1224704~20480] 58.c6f3131c ack+ondisk+write+known_if_redirected e91882) currently reached_pg
|
|
2015-10-26 07:53:06.419932 7f55417cf700 0 log_channel(cluster) log [WRN] : 67 slow requests, 12 included below; oldest blocked for > 20146.568552 secs
|
|
2015-10-26 07:53:06.419937 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 31.547642 seconds old, received at 2015-10-26 07:52:34.872234: osd_op(client.23189310.0:4268275 rbd_data.622b633d1b58ba.0000000000016a00 [set-alloc-hint object_size 4194304 write_size 4194304,write 1282048~4096] 58.c6f3131c ack+ondisk+write+known_if_redirected e91882) currently reached_pg
|
|
2015-10-26 07:53:06.419942 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.886655 seconds old, received at 2015-10-26 07:52:35.533220: osd_op(client.23188857.0:2342145 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91760) currently reached_pg
|
|
2015-10-26 07:53:06.419945 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 31.800247 seconds old, received at 2015-10-26 07:52:34.619629: osd_op(client.23209583.0:4018450 rbd_data.419e2e2ae8944a.0000000000000900 [set-alloc-hint object_size 4194304 write_size 4194304,write 2108928~16384] 58.2b966d4b ack+ondisk+write+known_if_redirected e91858) currently reached_pg
|
|
2015-10-26 07:53:06.419950 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 31.798462 seconds old, received at 2015-10-26 07:52:34.621413: osd_op(client.23189310.0:4268274 rbd_data.622b633d1b58ba.0000000000016a00 [set-alloc-hint object_size 4194304 write_size 4194304,write 1261568~12288] 58.c6f3131c ack+ondisk+write+known_if_redirected e91882) currently reached_pg
|
|
2015-10-26 07:53:06.419953 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 31.422317 seconds old, received at 2015-10-26 07:52:34.997559: osd_op(client.31349106.0:63988 rbd_header.41415b2ae8944a [watch ping cookie 139989157475056 gen 1] 58.5bdecd95 ondisk+write+known_if_redirected e91759) currently reached_pg
|
|
2015-10-26 07:53:09.747399 7f55417cf700 0 log_channel(cluster) log [WRN] : 56 slow requests, 7 included below; oldest blocked for > 20149.896008 secs
|
|
2015-10-26 07:53:09.747406 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 34.280824 seconds old, received at 2015-10-26 07:52:35.466507: osd_op(client.30072393.0:4735359 rb.0.4042a0.2ae8944a.000000001586 [sparse-read 1114112~4096] 58.e2bbb71c ack+read+known_if_redirected e91890) currently reached_pg
|
|
2015-10-26 07:53:09.747411 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 34.280104 seconds old, received at 2015-10-26 07:52:35.467227: osd_op(client.23189310.0:4268276 rbd_data.622b633d1b58ba.0000000000016a00 [set-alloc-hint object_size 4194304 write_size 4194304,write 1290240~61440] 58.c6f3131c ack+ondisk+write+known_if_redirected e91882) currently reached_pg
|
|
2015-10-26 07:53:09.747414 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 38.912067 seconds old, received at 2015-10-26 07:52:30.835264: osd_op(client.23188857.0:2342143 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91750) currently reached_pg
|
|
2015-10-26 07:53:09.747418 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 35.124807 seconds old, received at 2015-10-26 07:52:34.622524: osd_op(client.23209583.0:4018451 rbd_data.419e2e2ae8944a.0000000000000900 [set-alloc-hint object_size 4194304 write_size 4194304,write 2141696~4096] 58.2b966d4b ack+ondisk+write+known_if_redirected e91858) currently reached_pg
|
|
2015-10-26 07:53:09.747422 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 34.750242 seconds old, received at 2015-10-26 07:52:34.997090: osd_op(client.23209583.0:4018452 rbd_data.419e2e2ae8944a.0000000000000900 [set-alloc-hint object_size 4194304 write_size 4194304,write 2170368~8192] 58.2b966d4b ack+ondisk+write+known_if_redirected e91858) currently reached_pg
|
|
2015-10-26 07:53:11.397914 7f55417cf700 0 log_channel(cluster) log [WRN] : 56 slow requests, 2 included below; oldest blocked for > 20151.546557 secs
|
|
2015-10-26 07:53:11.397918 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 39.343869 seconds old, received at 2015-10-26 07:52:32.054011: osd_op(client.23188857.0:2342144 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91755) currently reached_pg
|
|
2015-10-26 07:53:11.397922 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 40.333907 seconds old, received at 2015-10-26 07:52:31.063974: osd_op(client.23189310.0:4268272 rbd_data.622b633d1b58ba.0000000000016a00 [set-alloc-hint object_size 4194304 write_size 4194304,write 1179648~40960] 58.c6f3131c ack+ondisk+write+known_if_redirected e91882) currently reached_pg
|
|
2015-10-26 07:53:14.301927 7f55417cf700 0 log_channel(cluster) log [WRN] : 54 slow requests, 9 included below; oldest blocked for > 20154.450557 secs
|
|
2015-10-26 07:53:14.301934 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 241.132882 seconds old, received at 2015-10-26 07:49:13.168999: osd_op(client.23912404.0:2205173 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91723) currently reached_pg
|
|
2015-10-26 07:53:14.301939 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 241.138964 seconds old, received at 2015-10-26 07:49:13.162916: osd_op(client.23188857.0:2342137 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91720) currently reached_pg
|
|
2015-10-26 07:53:14.301943 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 241.126364 seconds old, received at 2015-10-26 07:49:13.175517: osd_op(client.23188857.0:2342140 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91735) currently reached_pg
|
|
2015-10-26 07:53:14.301947 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 241.125409 seconds old, received at 2015-10-26 07:49:13.176472: osd_op(client.23912404.0:2205175 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91733) currently reached_pg
|
|
2015-10-26 07:53:14.301950 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 241.130958 seconds old, received at 2015-10-26 07:49:13.170923: osd_op(client.23188857.0:2342138 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91725) currently reached_pg
|
|
2015-10-26 07:53:15.684970 7f55417cf700 0 log_channel(cluster) log [WRN] : 58 slow requests, 7 included below; oldest blocked for > 20155.833605 secs
|
|
2015-10-26 07:53:15.684977 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 242.525513 seconds old, received at 2015-10-26 07:49:13.159415: osd_op(client.23188857.0:2342136 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91717) currently reached_pg
|
|
2015-10-26 07:53:15.684981 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 32.713520 seconds old, received at 2015-10-26 07:52:42.971409: osd_op(client.23189310.0:4268277 rbd_data.622b633d1b58ba.0000000000016a00 [set-alloc-hint object_size 4194304 write_size 4194304,write 1355776~24576] 58.c6f3131c ack+ondisk+write+known_if_redirected e91882) currently reached_pg
|
|
2015-10-26 07:53:15.684984 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 242.513283 seconds old, received at 2015-10-26 07:49:13.171645: osd_op(client.23188857.0:2342139 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91730) currently reached_pg
|
|
2015-10-26 07:53:15.684986 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.847983 seconds old, received at 2015-10-26 07:52:44.836946: osd_op(client.23188857.0:2342146 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91765) currently reached_pg
|
|
2015-10-26 07:53:15.684988 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 242.525014 seconds old, received at 2015-10-26 07:49:13.159914: osd_op(client.23912404.0:2205172 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91719) currently reached_pg
|
|
2015-10-26 07:53:17.144404 7f55417cf700 0 log_channel(cluster) log [WRN] : 58 slow requests, 2 included below; oldest blocked for > 20157.293043 secs
|
|
2015-10-26 07:53:17.144412 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 243.972483 seconds old, received at 2015-10-26 07:49:13.171883: osd_op(client.23912404.0:2205174 rbd_header.16cdf892ae8944a [watch ping cookie 140387649913536 gen 1] 58.6ea0fda5 ondisk+write+known_if_redirected e91731) currently reached_pg
|
|
2015-10-26 07:53:17.144417 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 243.984343 seconds old, received at 2015-10-26 07:49:13.160024: osd_op(client.29183509.0:3545779 rbd_data.414aa92ae8944a.00000000000056a2 [sparse-read 302592~4096] 58.9c107f4b ack+read+known_if_redirected e91732) currently reached_pg
|
|
2015-10-26 07:53:18.642715 7f54ffe61700 0 log_channel(cluster) log [INF] : 87.65s0 restarting backfill on osd.30(1) from (0'0,0'0] MAX to 89449'28302
|
|
2015-10-26 07:53:19.212167 7f55417cf700 0 log_channel(cluster) log [WRN] : 59 slow requests, 1 included below; oldest blocked for > 20159.360817 secs
|
|
2015-10-26 07:53:19.212198 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.456958 seconds old, received at 2015-10-26 07:52:48.755182: osd_op(client.23189310.0:4268278 rbd_data.622b633d1b58ba.0000000000016a00 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~20480] 58.c6f3131c ack+ondisk+write+known_if_redirected e91882) currently reached_pg
|
|
2015-10-26 07:53:19.225901 7f54ffe61700 0 log_channel(cluster) log [INF] : 87.53s0 restarting backfill on osd.28(4) from (0'0,0'0] MAX to 89445'28379
|
|
2015-10-26 07:53:19.229935 7f54ff660700 0 log_channel(cluster) log [INF] : 87.3as1 restarting backfill on osd.23(0) from (0'0,0'0] MAX to 89446'28288
|
|
2015-10-26 07:53:20.927150 7f55417cf700 0 log_channel(cluster) log [WRN] : 61 slow requests, 2 included below; oldest blocked for > 20161.075789 secs
|
|
2015-10-26 07:53:20.927158 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.711073 seconds old, received at 2015-10-26 07:52:50.216040: osd_op(client.23189310.0:4268279 rbd_data.622b633d1b58ba.0000000000016a00 [set-alloc-hint object_size 4194304 write_size 4194304,write 1409024~12288] 58.c6f3131c ack+ondisk+write+known_if_redirected e91882) currently reached_pg
|
|
2015-10-26 07:53:20.927163 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.710560 seconds old, received at 2015-10-26 07:52:50.216552: osd_op(client.23188857.0:2342147 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91770) currently reached_pg
|
|
2015-10-26 07:53:21.121796 7f54ff660700 0 log_channel(cluster) log [INF] : 87.7cs0 restarting backfill on osd.8(5) from (0'0,0'0] MAX to 89445'28309
|
|
2015-10-26 07:53:21.274404 7f54ff660700 0 log_channel(cluster) log [INF] : 87.6es0 restarting backfill on osd.15(6) from (0'0,0'0] MAX to 90157'28227
|
|
2015-10-26 07:53:24.178271 7f55417cf700 0 log_channel(cluster) log [WRN] : 57 slow requests, 3 included below; oldest blocked for > 20164.326910 secs
|
|
2015-10-26 07:53:24.178276 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 60.941193 seconds old, received at 2015-10-26 07:52:23.237040: osd_op(client.23188857.0:2342141 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91740) currently reached_pg
|
|
2015-10-26 07:53:24.178290 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 60.935121 seconds old, received at 2015-10-26 07:52:23.243112: osd_op(client.28525233.0:1290300 rbd_data.41609374b0dc51.00000000000004c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 2137600~4096] 58.b50d831c ack+ondisk+write+known_if_redirected e91884) currently reached_pg
|
|
2015-10-26 07:53:24.178292 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 60.934986 seconds old, received at 2015-10-26 07:52:23.243247: osd_op(client.23188857.0:2342142 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91745) currently reached_pg
|
|
2015-10-26 07:53:24.865618 7f54ffe61700 0 log_channel(cluster) log [INF] : 81.25d restarting backfill on osd.15 from (0'0,0'0] MAX to 89443'25956
|
|
2015-10-26 07:53:26.179330 7f55417cf700 0 log_channel(cluster) log [WRN] : 53 slow requests, 2 included below; oldest blocked for > 20166.327975 secs
|
|
2015-10-26 07:53:26.179340 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 60.583063 seconds old, received at 2015-10-26 07:52:25.596236: osd_op(client.29173392.0:807046 rbd_data.ca5da52ae8944a.00000000000000a6 [sparse-read 2732032~16384] 58.58fa6d3e ack+read+known_if_redirected e91886) currently reached_pg
|
|
2015-10-26 07:53:26.179343 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 60.909583 seconds old, received at 2015-10-26 07:52:25.269715: osd_op(client.30611677.0:40310 rbd_data.dd81e52ae8944a.0000000000005250 [sparse-read 48640~4096] 58.f0d3a953 ack+read+known_if_redirected e91877) currently reached_pg
|
|
2015-10-26 07:53:29.392724 7f55417cf700 0 log_channel(cluster) log [WRN] : 44 slow requests, 2 included below; oldest blocked for > 20169.541370 secs
|
|
2015-10-26 07:53:29.392729 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 60.501133 seconds old, received at 2015-10-26 07:52:28.891560: osd_op(client.23211368.0:41927684 rbd_data.447e1474b0dc51.0000000000004327 [sparse-read 2760192~16384] 58.e9ffea0b ack+read+known_if_redirected e91887) currently reached_pg
|
|
2015-10-26 07:53:29.392733 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.154127 seconds old, received at 2015-10-26 07:52:59.238566: osd_op(client.23189310.0:4268280 rbd_data.622b633d1b58ba.0000000000016a00 [set-alloc-hint object_size 4194304 write_size 4194304,write 1429504~4096] 58.c6f3131c ack+ondisk+write+known_if_redirected e91882) currently reached_pg
|
|
2015-10-26 07:53:29.568501 7f54ff660700 0 log_channel(cluster) log [INF] : 58.44 restarting backfill on osd.15 from (0'0,0'0] MAX to 90122'2338536
|
|
2015-10-26 07:53:29.887869 7f54ffe61700 0 log_channel(cluster) log [INF] : 81.aa5 restarting backfill on osd.13 from (0'0,0'0] MAX to 90045'23139
|
|
2015-10-26 07:53:29.905617 7f54ff660700 0 log_channel(cluster) log [INF] : 58.13e restarting backfill on osd.15 from (0'0,0'0] MAX to 90122'4858667
|
|
2015-10-26 07:53:29.917378 7f54ff660700 0 log_channel(cluster) log [INF] : 81.5b6 restarting backfill on osd.14 from (0'0,0'0] MAX to 90122'21678
|
|
2015-10-26 07:53:31.394775 7f55417cf700 0 log_channel(cluster) log [WRN] : 41 slow requests, 3 included below; oldest blocked for > 20171.543394 secs
|
|
2015-10-26 07:53:31.394782 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 60.559471 seconds old, received at 2015-10-26 07:52:30.835246: osd_op(client.23211368.0:41927685 rbd_data.447e1474b0dc51.0000000000004327 [sparse-read 2780672~12288] 58.e9ffea0b ack+read+known_if_redirected e91887) currently reached_pg
|
|
2015-10-26 07:53:31.394787 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 60.559453 seconds old, received at 2015-10-26 07:52:30.835264: osd_op(client.23188857.0:2342143 rbd_header.435cba2ae8944a [watch ping cookie 139630057885136 gen 3] 58.a9e6651c ondisk+write+known_if_redirected e91750) currently reached_pg
|
|
2015-10-26 07:53:31.394791 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 60.330743 seconds old, received at 2015-10-26 07:52:31.063974: osd_op(client.23189310.0:4268272 rbd_data.622b633d1b58ba.0000000000016a00 [set-alloc-hint object_size 4194304 write_size 4194304,write 1179648~40960] 58.c6f3131c ack+ondisk+write+known_if_redirected e91882) currently waiting for subops from 13,16
|
|
2015-10-26 07:53:43.720998 7f55417cf700 0 log_channel(cluster) log [WRN] : 15 slow requests, 3 included below; oldest blocked for > 20183.869636 secs
|
|
2015-10-26 07:53:43.721005 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 33.967393 seconds old, received at 2015-10-26 07:53:09.753566: osd_op(client.23199808.0:24351537 rbd_data.41af532ae8944a.000000000000016e [sparse-read 105984~8192] 58.33759c44 ack+read+known_if_redirected e91898) currently reached_pg
|
|
2015-10-26 07:53:43.721009 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 33.971005 seconds old, received at 2015-10-26 07:53:09.749954: osd_op(client.23199808.0:24351536 rbd_data.41af532ae8944a.000000000000016e [sparse-read 93696~8192] 58.33759c44 ack+read+known_if_redirected e91898) currently reached_pg
|
|
2015-10-26 07:53:43.721011 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 35.487989 seconds old, received at 2015-10-26 07:53:08.232969: osd_op(client.23199808.0:24351535 rbd_data.41af532ae8944a.000000000000016e [sparse-read 85504~4096] 58.33759c44 ack+read+known_if_redirected e91898) currently reached_pg
|
|
2015-10-26 07:53:47.765107 7f55417cf700 0 log_channel(cluster) log [WRN] : 15 slow requests, 1 included below; oldest blocked for > 20187.913749 secs
|
|
2015-10-26 07:53:47.765113 7f55417cf700 0 log_channel(cluster) log [WRN] : slow request 30.343732 seconds old, received at 2015-10-26 07:53:17.421340: osd_op(client.23186067.0:17661775 rbd_data.437e7e2ae8944a.0000000000004e93 [set-alloc-hint object_size 4194304 write_size 4194304,write 2330112~49152] 58.9dc4653e ack+ondisk+write+known_if_redirected e91901) currently waiting for subops from 13,15,17,20
|
|
|