2014-05-19 18:33:46.468704 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:46.468763 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 1.05 >= max 0.5 = no, load too high 2014-05-19 18:33:46.468776 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:33:46.468779 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:46.468779 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 69.1928 < min (86400 seconds) 2014-05-19 18:33:46.468799 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:33:46.468802 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:46.468804 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:46.767020 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:33:46.767778 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:46.939529 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:33:47.133410 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:47.133513 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:47.333684 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.333816 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.334034 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.334596 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.334790 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.334968 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.335064 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.347843 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.347867 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.355255 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.360170 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.381881 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:33:47.385335 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.385361 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.403114 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.403137 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.411315 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.411337 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.411358 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.411379 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:33:47.411682 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:47.468873 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:47.468890 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:33:47.468894 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:33:47.468898 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:33:47.468906 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:33:47.468910 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:47.468912 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:47.468950 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:48.233898 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:48.233963 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:48.469001 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:48.469021 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:33:48.469026 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:48.469028 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:48.469069 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:48.625690 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:33:48.625709 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:33:48.626278 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:48.626350 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:48.626428 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:48.626494 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:48.626528 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:48.626574 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:48.626642 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:48.626681 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:48.626707 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:48.626735 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:48.626760 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:48.626783 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:48.626809 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:48.626834 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:48.626858 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:48.626881 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:48.767120 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:33:48.767860 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:48.939624 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:33:49.063679 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:49.063776 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:49.170266 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:49.170364 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:49.333770 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.333894 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.334119 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.334680 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.334870 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.335032 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.335125 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.347944 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.347999 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.355341 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.360267 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.381963 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:33:49.385437 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.385461 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.403214 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.403241 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.411414 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.411438 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.411457 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.411476 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:33:49.411747 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:49.469116 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:49.469164 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.96 >= max 0.5 = no, load too high 2014-05-19 18:33:49.469178 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:33:49.469184 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:49.469181 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 72.1932 < min (86400 seconds) 2014-05-19 18:33:49.469199 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:33:49.469202 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:49.469204 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:49.759196 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:49.759329 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:49.952562 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:49.952677 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:50.095710 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:50.095761 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:50.164225 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:50.164273 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:50.270691 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:50.270751 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:50.469273 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:50.469324 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:50.469320 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.96 >= max 0.5 = no, load too high 2014-05-19 18:33:50.469334 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:33:50.469337 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 73.1934 < min (86400 seconds) 2014-05-19 18:33:50.469355 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:33:50.469358 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:50.469360 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:50.767199 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:33:50.767939 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:50.939717 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:33:50.996450 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:50.996529 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:51.052935 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:51.052995 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:51.142785 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:51.142856 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:51.196134 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:51.196196 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:51.333850 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.333975 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.334185 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.334761 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.334947 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.335095 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.335185 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.348026 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.348082 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.355423 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.360364 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.382045 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:33:51.385527 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.385548 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.403313 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.403337 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.411515 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.411536 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.411550 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.411569 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:33:51.411810 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:51.469433 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:51.469454 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:33:51.469459 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:51.469461 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:51.469503 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:52.469535 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:52.469552 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:33:52.469556 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:33:52.469560 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:33:52.469595 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.96 >= max 0.5 = no, load too high 2014-05-19 18:33:52.469607 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:33:52.469611 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:52.469611 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 75.1936 < min (86400 seconds) 2014-05-19 18:33:52.469630 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:33:52.469633 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:52.469635 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:52.726381 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:33:52.726399 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:33:52.726855 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:52.726910 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:52.726992 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:52.727033 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:52.727067 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:52.727110 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:52.727157 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:52.727208 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:52.727253 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:52.727280 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:52.727313 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:52.727340 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:52.727366 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:52.727388 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:52.727410 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:52.727441 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:52.767295 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:33:52.768029 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:52.939813 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:33:53.226945 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:33:53.226963 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:33:53.227387 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:53.227548 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:53.227586 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:53.227616 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:53.227638 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:53.227676 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:53.227720 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:53.227762 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:53.227817 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:53.227842 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:53.227870 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:53.227892 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:53.227914 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:53.227937 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:53.227959 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:53.227981 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:53.333922 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.334058 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.334255 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.334843 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.335028 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.335158 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.335247 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.348109 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.348159 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.355514 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.360461 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.382128 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:33:53.385628 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.385653 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.403412 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.403436 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.411618 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.411642 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.411696 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.411729 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:33:53.411886 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:53.469708 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:53.469729 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:33:53.469734 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:53.469736 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:53.469777 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:53.859792 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:53.859861 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:54.134566 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:54.134629 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:54.360337 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:54.360441 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:54.469811 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:54.469832 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:33:54.469836 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:54.469839 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:54.469862 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:54.767393 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:33:54.768109 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:54.939896 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:33:55.096901 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:55.097015 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:55.243281 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:55.243363 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:55.334011 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.334137 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.334337 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.334925 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.335110 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.335222 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.335307 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.348193 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.348236 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.355610 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.360559 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.382209 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:33:55.385728 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.385753 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.403512 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.403539 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.411719 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.411745 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.411779 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.411832 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:33:55.411967 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:55.469908 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:55.469956 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.89 >= max 0.5 = no, load too high 2014-05-19 18:33:55.469969 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:33:55.469976 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:55.469972 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 78.194 < min (86400 seconds) 2014-05-19 18:33:55.469991 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:33:55.469994 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:55.469996 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:55.597428 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:55.597479 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:55.743811 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:55.744179 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:56.064671 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:56.064747 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:56.127496 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:33:56.127514 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:33:56.127985 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:56.128064 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:56.128154 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:56.128190 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:56.128219 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:56.128262 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:33:56.128307 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:56.128365 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:56.128410 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:33:56.128453 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:33:56.128501 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:56.128543 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:56.128611 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:56.128653 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:56.128701 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:56.128744 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:56.171014 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:56.171077 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:33:56.470064 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:56.470086 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:33:56.470090 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:56.470093 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:56.470133 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:56.767490 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:33:56.768189 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:56.939975 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:33:56.953292 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:56.953357 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:33:57.096451 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:57.096555 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:33:57.260886 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:57.260977 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:33:57.334111 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.334215 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.334415 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.335008 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.335193 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.335286 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.335368 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.348282 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.348314 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.355706 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.360657 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.382295 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:33:57.385829 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.385883 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.403611 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.403633 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.411818 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.411841 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.411856 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.411897 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:33:57.412050 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:57.470162 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:57.470179 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:33:57.470183 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:33:57.470187 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:33:57.470221 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.89 >= max 0.5 = no, load too high 2014-05-19 18:33:57.470234 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:33:57.470239 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:57.470237 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 80.1943 < min (86400 seconds) 2014-05-19 18:33:57.470257 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:33:57.470260 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:57.470262 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:58.235033 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:58.235086 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:58.470333 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:58.470381 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.81 >= max 0.5 = no, load too high 2014-05-19 18:33:58.470393 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:33:58.470401 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:33:58.470396 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 81.1944 < min (86400 seconds) 2014-05-19 18:33:58.470415 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:33:58.470418 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:58.470420 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:58.735578 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:58.735644 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:33:58.767588 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:33:58.768268 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:58.940054 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:33:59.334207 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.334271 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.334463 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.335090 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.335272 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.335350 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.335430 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.348365 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.348393 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.355803 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.360754 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.382372 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:33:59.385926 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.385968 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.403708 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.403731 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.411918 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.411943 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.411961 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.411981 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:33:59.412117 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:33:59.470499 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:33:59.470520 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:33:59.470524 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:33:59.470527 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:33:59.470566 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:00.165123 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:00.165168 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:00.271537 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:00.271598 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:00.470612 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:00.470632 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:00.470637 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:00.470639 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:00.470679 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:00.665569 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:00.665641 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:00.767686 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:00.768347 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:00.772086 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:00.772256 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:00.828027 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:00.828046 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:00.828485 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:00.828530 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:00.828564 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:00.828637 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:00.828665 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:00.828724 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:00.828775 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:00.828815 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:00.828858 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:00.828878 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:00.828903 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:00.828924 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:00.828950 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:00.828973 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:00.828993 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:00.829036 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:00.897933 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:00.898041 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:00.940137 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:01.044462 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:01.044525 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:01.053703 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:01.053754 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:01.196756 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:01.196882 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:01.334303 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.334347 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.334541 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.335174 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.335357 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.335421 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.335490 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.348450 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.348472 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.355901 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.360851 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.382468 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:01.386022 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.386047 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.403807 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.403833 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.412004 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.412024 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.412041 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.412069 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:01.412181 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:01.470725 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:01.470746 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:01.470751 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:01.470753 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:01.470793 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:01.554151 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:01.554216 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:01.697229 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:01.697312 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:01.928562 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:01.928601 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:01.929055 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:01.929103 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:01.929175 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:01.929211 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:01.929256 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:01.929311 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:01.929342 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:01.929457 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:01.929493 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:01.929514 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:01.929540 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:01.929559 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:01.929580 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:01.929600 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:01.929634 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:01.929656 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:01.961540 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:01.961660 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:02.470837 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:02.470853 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:34:02.470857 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:34:02.470861 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:34:02.470869 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:02.470873 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:02.470875 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:02.470887 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:02.598451 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:02.598525 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:02.745087 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:02.745259 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:02.767784 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:02.768430 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:02.940211 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:03.062075 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:03.062141 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:03.334400 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.334425 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.334612 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.335255 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.335438 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.335483 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.335550 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.348551 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.348597 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.355998 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.360932 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.382548 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:03.386105 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.386128 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.403906 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.403929 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.412086 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.412108 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.412126 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.412156 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:03.412245 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:03.470962 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:03.471010 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.75 >= max 0.5 = no, load too high 2014-05-19 18:34:03.471023 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:34:03.471030 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:03.471026 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 86.1951 < min (86400 seconds) 2014-05-19 18:34:03.471046 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:03.471049 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:03.471051 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:03.699090 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:03.699146 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:03.845673 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:03.845725 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:04.035985 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:04.036087 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:04.471138 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:04.471158 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:04.471163 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:04.471165 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:04.471205 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:04.767881 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:04.768511 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:04.940278 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:05.334454 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.334497 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.334687 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.335338 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.335521 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.335549 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.335613 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.348647 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.348675 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.356095 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.361016 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.382626 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:05.386194 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.386214 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.404007 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.404033 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.412171 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.412191 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.412208 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.412229 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:05.412310 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:05.471251 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:05.471271 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:05.471277 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:05.471279 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:05.471308 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:05.736459 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:05.736520 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:05.966079 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:05.966137 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:06.029141 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:06.029159 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43968 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:06.029685 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:06.029755 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:06.029792 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:06.029813 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:06.029832 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:06.029851 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:06.029944 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:06.029986 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:06.030010 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:06.030031 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:06.030060 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:06.030087 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:06.030113 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:06.030136 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:06.030156 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:06.030178 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:06.072340 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:06.072475 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:06.471365 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:06.471385 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:06.471390 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:06.471392 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:06.471432 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:06.767956 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:06.768591 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:06.837096 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:06.837154 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:06.854409 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:06.854480 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:06.940364 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:06.997514 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:06.997616 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:07.162655 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:07.162743 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:07.334548 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.334570 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.334764 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.335427 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.335609 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.335631 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.335675 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.348747 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.348769 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.356182 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.361097 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.382724 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:07.386277 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.386299 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.404107 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.404135 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.412253 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.412272 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.412290 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.412316 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:07.412376 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:07.471478 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:07.471495 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:34:07.471499 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:34:07.471502 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:34:07.471537 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.75 >= max 0.5 = no, load too high 2014-05-19 18:34:07.471549 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:34:07.471553 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:07.471552 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 90.1956 < min (86400 seconds) 2014-05-19 18:34:07.471572 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:07.471575 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:07.471577 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:07.666579 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:07.666650 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:07.772801 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:07.772853 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:07.799572 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:07.799663 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:07.946207 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:07.946271 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:08.471662 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:08.471722 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:08.471719 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.69 >= max 0.5 = no, load too high 2014-05-19 18:34:08.471750 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:34:08.471756 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 91.1958 < min (86400 seconds) 2014-05-19 18:34:08.471780 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:08.471786 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:08.471789 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:08.554749 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:08.554807 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:08.697943 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:08.698005 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:08.766985 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:08.767088 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:08.768038 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:08.768672 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:08.873180 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:08.873266 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:08.940455 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:09.334630 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.334658 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.334839 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.335509 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.335687 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.335737 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.335753 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.348849 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.348873 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.356264 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.361193 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.382820 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:09.386361 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.386383 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.404208 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.404233 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.412347 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.412368 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.412388 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.412400 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:09.412444 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:09.471877 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:09.471898 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:09.471903 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:09.471905 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:09.471935 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:09.655168 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:09.655272 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:09.798345 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:09.798439 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:10.471988 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:10.472036 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.69 >= max 0.5 = no, load too high 2014-05-19 18:34:10.472049 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:34:10.472056 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:10.472052 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 93.1961 < min (86400 seconds) 2014-05-19 18:34:10.472071 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:10.472074 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:10.472076 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:10.768118 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:10.768752 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:10.937562 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:10.937647 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:10.940543 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:11.334729 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.334786 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.334910 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.335590 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.335776 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.335804 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.335864 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.348949 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.348975 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.356350 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.361292 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.382914 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:11.386445 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.386466 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.404308 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.404331 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.412451 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.412475 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.412495 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.412510 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:11.412546 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:11.472162 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:11.472184 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:11.472188 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:11.472190 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:11.472230 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:11.929749 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:11.929767 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:11.930252 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:11.930310 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:11.930355 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:11.930389 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:11.930414 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:11.930460 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:11.930483 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:11.930509 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:11.930575 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:11.930619 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:11.930646 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:11.930668 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:11.930694 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:11.930716 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:11.930737 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:11.930761 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:12.472272 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:12.472289 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:34:12.472293 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:34:12.472296 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:34:12.472304 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:12.472308 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:12.472310 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:12.472348 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:12.768196 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:12.768830 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:12.867414 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:12.867477 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:12.940636 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:12.973495 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:12.973557 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:13.030337 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:13.030355 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:13.030828 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:13.030887 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:13.030939 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:13.031038 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:13.031062 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:13.031110 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:13.031152 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:13.031177 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:13.031205 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:13.031230 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:13.031255 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:13.031277 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:13.031302 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:13.031324 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:13.031345 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:13.031367 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:13.063150 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:13.063235 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:13.334826 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.334865 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.334986 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.335672 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.335842 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.335905 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.335943 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.349032 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.349054 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.356445 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.361390 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.383009 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:13.386526 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.386550 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.404409 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.404470 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.412551 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.412599 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.412602 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:13.412617 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.412665 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:13.472396 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:13.472417 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:13.472422 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:13.472424 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:13.472464 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:13.699959 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:13.700042 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:13.755502 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:13.755613 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:13.846738 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:13.846862 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:13.898663 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:13.898750 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:14.163809 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:14.163921 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:14.472511 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:14.472559 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.63 >= max 0.5 = no, load too high 2014-05-19 18:34:14.472572 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:34:14.472592 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:14.472595 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 97.1966 < min (86400 seconds) 2014-05-19 18:34:14.472615 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:14.472618 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:14.472620 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:14.768292 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:14.768910 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:14.800433 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:14.800563 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:14.940722 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:14.947385 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:14.947545 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:15.334911 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.334940 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.335048 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.335755 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.335911 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.335991 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.336026 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.349128 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.349152 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.356541 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.361489 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.383104 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:15.386607 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.386630 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.404507 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.404547 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.412650 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.412674 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:15.412680 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.412701 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.412730 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:15.472705 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:15.472726 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:15.472731 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:15.472733 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:15.472773 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:16.472816 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:16.472837 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:16.472841 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:16.472844 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:16.472884 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:16.768347 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:16.768989 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:16.838093 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:16.838200 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:16.940812 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:17.335011 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.335035 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.335120 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.335852 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.335979 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.336080 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.336107 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.349211 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.349231 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.356638 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.361587 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.383198 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:17.386709 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.386762 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.404606 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.404625 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.412750 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.412750 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:17.412778 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.412795 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.412809 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:17.472915 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:17.472932 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:34:17.472936 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:34:17.472940 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:34:17.472975 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.63 >= max 0.5 = no, load too high 2014-05-19 18:34:17.472987 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:34:17.472991 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:17.472990 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 100.197 < min (86400 seconds) 2014-05-19 18:34:17.473030 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:17.473033 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:17.473035 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:17.938571 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:17.938633 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:18.473106 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:18.473154 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.58 >= max 0.5 = no, load too high 2014-05-19 18:34:18.473167 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:34:18.473174 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:18.473170 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 101.197 < min (86400 seconds) 2014-05-19 18:34:18.473188 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:18.473192 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:18.473193 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:18.767871 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:18.767924 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:18.768388 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:18.769070 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:18.873779 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:18.873861 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:18.930920 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:18.930939 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:18.931468 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:18.931504 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:18.931547 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:18.931585 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:18.931647 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:18.931695 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:18.931720 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:18.931746 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:18.931768 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:18.931811 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:18.931851 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:18.931873 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:18.931899 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:18.931922 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:18.931944 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:18.931970 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:18.940884 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:19.335104 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.335130 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.335192 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.335929 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.336037 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.336160 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.336186 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.349293 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.349314 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.356735 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.361685 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.383294 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:19.386806 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.386847 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.404707 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.404734 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.412854 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:19.412858 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.412928 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.412979 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.413036 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:19.473257 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:19.473276 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:19.473279 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:19.473281 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:19.473312 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:19.655766 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:19.655824 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:19.798900 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:19.798952 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:19.868375 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:19.868485 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:19.974185 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:19.974280 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:20.064421 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:20.064553 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:20.473358 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:20.473379 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:20.473384 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:20.473386 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:20.473426 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:20.700872 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:20.700993 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:20.756278 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:20.756333 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:20.768483 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:20.769148 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:20.847893 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:20.847995 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:20.899279 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:20.899324 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:20.940964 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:21.335205 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.335272 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.335294 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.336012 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.336103 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.336247 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.336268 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.349377 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.349397 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.356833 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.361782 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.383388 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:21.386904 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.386928 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.404788 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.404817 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.412963 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:21.412967 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.413005 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.413055 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.413112 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:21.473448 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:21.473466 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:21.473470 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:21.473472 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:21.473497 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:22.431535 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:22.431553 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:22.432085 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:22.432133 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:22.432170 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:22.432193 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:22.432283 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:22.432359 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:22.432433 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:22.432472 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:22.432498 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:22.432523 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:22.432551 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:22.432573 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:22.432638 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:22.432664 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:22.432686 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:22.432711 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:22.473542 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:22.473560 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:34:22.473566 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:34:22.473573 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:34:22.473584 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:22.473590 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:22.473590 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:22.473593 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:22.768597 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:22.769220 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:22.941041 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:23.335294 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.335351 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.335371 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.336092 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.336166 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.336328 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.336349 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.349459 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.349479 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.356914 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.361866 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.383482 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:23.387004 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.387031 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.404869 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.404897 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.413061 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:23.413065 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.413083 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.413123 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.413214 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:23.473681 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:23.473729 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.54 >= max 0.5 = no, load too high 2014-05-19 18:34:23.473742 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:34:23.473748 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:23.473745 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 106.198 < min (86400 seconds) 2014-05-19 18:34:23.473765 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:23.473768 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:23.473770 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:23.564999 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:23.565135 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:23.839003 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:23.839076 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:24.201407 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:24.201477 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:24.348437 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:24.348542 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:24.473857 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:24.473878 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:24.473883 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:24.473885 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:24.473925 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:24.768683 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:24.769298 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:24.941103 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:25.332092 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:25.332110 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:25.332554 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:25.332645 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:25.332682 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:25.332715 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:25.332793 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:25.332846 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:25.332885 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:25.332923 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:25.332946 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:25.332970 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:25.332995 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:25.333017 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:25.333039 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:25.333064 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:25.333086 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:25.333107 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:25.335373 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.335419 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.335450 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.336173 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.336234 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.336409 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.336430 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.349543 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.349563 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.356998 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.361949 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.383576 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:25.387103 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.387128 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.404949 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.404978 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.413145 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:25.413149 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.413167 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.413191 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.413285 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:25.473950 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:25.473997 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.54 >= max 0.5 = no, load too high 2014-05-19 18:34:25.474010 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:34:25.474017 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:25.474014 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 108.198 < min (86400 seconds) 2014-05-19 18:34:25.474031 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:25.474034 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:25.474036 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:25.768857 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:25.768931 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:25.874463 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:25.874533 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:26.474118 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:26.474139 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:26.474144 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:26.474146 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:26.474185 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:26.656501 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:26.656619 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:26.768765 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:26.769376 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:26.799549 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:26.799656 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:26.941184 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:27.101950 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:27.102019 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:27.249012 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:27.249115 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:27.335471 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.335496 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.335531 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.336261 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.336304 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.336491 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.336512 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.339459 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:27.339540 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:27.349626 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.349646 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.357083 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.362045 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.383670 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:27.387202 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.387226 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.405033 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.405060 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.413227 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:27.413231 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.413248 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.413264 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.413348 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:27.474221 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:27.474238 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:34:27.474242 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:34:27.474245 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:34:27.474271 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:27.474280 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.54 >= max 0.5 = no, load too high 2014-05-19 18:34:27.474292 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=0 2014-05-19 18:34:27.474296 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 110.198 < min (86400 seconds) 2014-05-19 18:34:27.474314 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:27.474317 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:27.474319 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:28.474397 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:28.474448 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:28.474445 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.49 < max 0.5 = yes 2014-05-19 18:34:28.474458 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:34:28.474461 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 111.198 < min (86400 seconds) 2014-05-19 18:34:28.474480 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:28.474483 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:28.474485 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:28.768811 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:28.769457 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:28.865663 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:28.865785 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:28.941256 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:29.269321 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:29.269379 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:29.335568 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.335607 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.335625 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.336358 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.336388 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.336600 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.336655 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.349719 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.349741 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.357165 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.362143 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.366095 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:29.366160 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:29.374801 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:29.374878 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:29.383767 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:29.387293 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.387312 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.405115 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.405142 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.413310 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:29.413314 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.413332 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.413349 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.413412 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:29.474571 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:29.474592 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:29.474597 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:29.474599 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:29.474639 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:30.156820 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:30.156886 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:30.239958 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:30.240011 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:30.299904 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:30.299968 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:30.474687 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:30.474708 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:30.474713 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:30.474715 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:30.474755 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:30.768878 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:30.769536 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:30.941331 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:31.232636 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:31.232655 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:31.233132 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:31.233180 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:31.233206 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:31.233240 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:31.233260 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:31.233309 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:31.233333 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:31.233353 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:31.233493 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:31.233532 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:31.233584 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:31.233627 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:31.233673 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:31.233720 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:31.233764 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:31.233808 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:31.335645 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.335683 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.335711 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.336441 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.336469 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.336681 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.336734 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.349801 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.349821 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.357254 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.362242 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.383861 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:31.387391 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.387417 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.405197 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.405225 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.413394 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:31.413398 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.413437 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.413456 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.413474 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:31.474788 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:31.474837 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.49 < max 0.5 = yes 2014-05-19 18:34:31.474849 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:34:31.474856 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:31.474853 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 114.199 < min (86400 seconds) 2014-05-19 18:34:31.474871 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:31.474875 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:31.474876 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:31.733191 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:31.733209 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:31.733652 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:31.733703 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:31.733749 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:31.733804 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:31.733860 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:31.733900 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:31.733939 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:31.733971 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:31.733992 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:31.734018 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:31.734043 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:31.734066 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:31.734091 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:31.734113 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:31.734135 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:31.734158 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:32.169810 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:32.169871 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:32.474958 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:32.474977 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:34:32.474982 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:34:32.474988 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:34:32.474999 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:32.475005 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:32.475009 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:32.475040 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:32.768935 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:32.769609 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:32.941400 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:33.002438 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:33.002515 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:33.057110 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:33.057215 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:33.149533 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:33.149624 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:33.335742 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.335761 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.335780 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.336526 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.336546 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.336764 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.336818 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.349900 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.349926 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.357335 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.362341 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.383957 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:33.387471 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.387497 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.405281 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.405303 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.413495 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:33.413499 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.413517 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.413533 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.413559 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:33.475081 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:33.475102 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:33.475107 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:33.475109 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:33.475132 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:33.502965 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:33.503037 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:33.650187 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:33.650274 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:34.475173 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:34.475194 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:34.475198 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:34.475200 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:34.475241 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:34.666646 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:34.666697 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:34.675201 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:34.675284 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:34.769012 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:34.769687 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:34.941481 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:35.175694 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:35.175767 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:35.335833 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.335859 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.335882 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.336613 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.336637 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.336845 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.336903 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.349998 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.350024 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.357420 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.362438 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.384053 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:35.387570 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.387595 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.405366 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.405385 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.413579 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:35.413583 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.413601 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.413620 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.413635 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:35.475272 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:35.475292 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:35.475297 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:35.475299 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:35.475323 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:35.600333 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:35.600400 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:35.833726 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:35.833744 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:35.834252 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:35.834307 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:35.834378 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:35.834419 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:35.834449 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:35.834551 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:35.834583 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:35.834603 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:35.834643 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:35.834677 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:35.834703 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:35.834725 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:35.834751 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:35.834773 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:35.834795 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:35.834817 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:36.100815 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:36.100899 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:36.140435 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:36.140547 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:36.475365 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:36.475386 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:36.475391 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:36.475393 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:36.475433 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:36.641015 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:36.641118 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:36.769096 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:36.769766 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:36.934302 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:36.934320 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:36.934785 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:36.934852 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:36.934921 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:36.934954 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:36.935003 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:36.935027 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:36.935064 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:36.935152 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:36.935177 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:36.935197 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:36.935222 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:36.935244 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:36.935266 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:36.935288 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:36.935310 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:36.935332 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:36.941560 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:37.335929 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.335951 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.335987 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.336712 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.336736 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.336926 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.336987 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.350096 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.350122 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.357502 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.362536 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.384147 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:37.387652 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.387675 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.405458 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.405483 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.413672 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:37.413676 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.413703 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.413724 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.413773 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:37.475478 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:37.475494 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:34:37.475498 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:34:37.475502 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:34:37.475509 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:37.475513 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:37.475515 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:37.475528 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:37.603352 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:37.603472 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:37.750738 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:37.750854 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:38.070248 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:38.070297 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:38.475600 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:38.475621 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:38.475625 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:38.475627 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:38.475651 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:38.570744 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:38.570842 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:38.703809 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:38.703907 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:38.769179 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:38.769847 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:38.851307 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:38.851465 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:38.941644 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:38.957318 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:38.957441 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:39.336022 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.336049 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.336062 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.336797 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.336820 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.337007 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.337069 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.350185 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.350206 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.357583 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.362634 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.384244 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:39.387731 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.387754 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.405540 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.405561 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.413774 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:39.413778 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.413801 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.413825 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.413855 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:39.457842 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:39.457941 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:39.475715 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:39.475736 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:39.475741 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:39.475743 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:39.475782 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:39.967178 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:39.967302 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:40.434864 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:40.434883 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:40.435352 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:40.435398 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:40.435503 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:40.435536 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:40.435568 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:40.435590 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:40.435614 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:40.435635 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:40.435654 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:40.435691 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:40.435716 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:40.435753 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:40.435779 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:40.435800 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:40.435822 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:40.435844 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:40.475813 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:40.475835 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:40.475839 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:40.475842 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:40.475842 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:40.476134 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:40.476202 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:40.741468 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:40.741573 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:40.769275 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:40.769930 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:40.941732 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:41.336120 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.336179 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.336230 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.336883 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.336906 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.337090 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.337146 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.350285 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.350311 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.357673 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.362717 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.384339 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:41.387802 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.387832 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.401177 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:41.401258 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:41.405634 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.405659 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.413874 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:41.413878 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.413897 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.413917 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.413931 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:41.475927 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:41.475947 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:41.475952 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:41.475954 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:41.475994 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:41.842144 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:41.842238 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:42.204254 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:42.204341 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:42.351842 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:42.351949 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:42.476025 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:42.476042 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:34:42.476046 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:34:42.476050 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:34:42.476057 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:42.476061 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:42.476063 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:42.476101 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:42.671227 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:42.671329 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:42.769373 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:42.770010 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:42.941826 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:43.336212 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.336280 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.336308 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.336967 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.336994 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.337172 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.337247 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.350384 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.350409 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.357768 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.362799 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.384433 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:43.387873 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.387901 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.405734 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.405757 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.413969 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:43.413976 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.413994 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.414015 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.414034 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:43.476147 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:43.476168 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:43.476173 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:43.476175 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:43.476215 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:43.558406 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:43.558521 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:43.771752 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:43.771810 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:44.476261 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:44.476283 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:44.476290 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:44.476294 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:44.476320 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:44.658868 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:44.658949 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:44.769470 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:44.770090 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:44.941916 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:45.336272 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.336338 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.336359 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.337052 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.337081 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.337255 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.337318 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.342735 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:45.342833 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:45.350484 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.350507 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.357866 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.362897 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.384530 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:45.387971 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.388004 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.405835 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.405861 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.414065 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:45.414071 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.414084 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.414097 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.414110 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:45.476381 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:45.476402 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:45.476407 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:45.476409 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:45.476448 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:45.735409 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:45.735427 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:45.735861 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:45.735903 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:45.736038 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:45.736098 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:45.736146 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:45.736195 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:45.736241 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:45.736284 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:45.736326 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:45.736371 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:45.736420 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:45.736463 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:45.736508 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:45.736551 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:45.736623 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:45.736667 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:45.776368 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:45.776496 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:45.867739 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:45.867808 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:46.368184 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:46.368249 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:46.476491 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:46.476511 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:46.476516 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:46.476518 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:46.476559 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:46.701487 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:46.701606 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:46.769563 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:46.770168 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:46.942011 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:47.272189 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:47.272291 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:47.336364 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.336412 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.336442 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.337149 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.337173 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.337339 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.337379 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.350583 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.350608 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.357963 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.362995 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.384609 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:47.388068 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.388097 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.405921 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.405940 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.414162 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:47.414168 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.414187 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.414238 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.414289 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:47.476609 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:47.476627 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:34:47.476631 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:34:47.476635 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:34:47.476643 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:47.476646 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:47.476648 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:47.476686 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:47.504741 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:47.504817 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:47.652469 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:47.652640 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:48.159259 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:48.159336 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:48.476721 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:48.476743 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:48.476747 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:48.476750 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:48.476789 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:48.635958 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:48.635976 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:48.636397 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:48.636431 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:48.636468 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:48.636631 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:48.636703 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:48.636770 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:48.636797 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:48.636823 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:48.636842 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:48.636880 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:48.636904 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:48.636924 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:48.636948 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:48.636967 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:48.636990 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:48.637011 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:48.769650 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:48.770241 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:48.942101 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:49.268719 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:49.268882 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:49.336441 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.336480 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.336539 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.337247 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.337304 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.337423 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.337477 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.350683 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.350739 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.358061 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.363093 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.384690 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:49.388168 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.388194 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.406004 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.406024 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.414260 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:49.414264 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.414301 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.414332 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.414365 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:49.476824 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:49.476872 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.35 < max 0.5 = yes 2014-05-19 18:34:49.476885 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:34:49.476893 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:49.476888 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 132.201 < min (86400 seconds) 2014-05-19 18:34:49.476907 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:49.476910 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:49.476912 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:50.405096 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:50.405201 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:50.476988 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:50.477010 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:50.477015 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:50.477017 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:50.477056 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:50.552998 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:50.553119 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:50.643172 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:50.643223 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:50.769740 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:50.770320 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:50.936506 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:50.936525 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:50.937103 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:50.937173 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:50.937215 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:50.937266 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:50.937309 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:50.937338 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:50.937378 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:50.937403 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:50.937423 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:50.937442 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:50.937464 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:50.937487 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:50.937507 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:50.937528 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:50.937549 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:50.937571 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:50.942169 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:51.336531 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.336563 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.336621 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.337330 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.337378 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.337498 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.337545 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.350780 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.350822 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.358157 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.363190 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.384772 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:51.388268 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.388292 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.406104 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.406130 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.414368 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:51.414371 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.414408 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.414432 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.414449 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:51.477104 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:51.477152 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.35 < max 0.5 = yes 2014-05-19 18:34:51.477165 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:34:51.477172 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:51.477169 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 134.201 < min (86400 seconds) 2014-05-19 18:34:51.477187 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:51.477190 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:51.477192 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:51.676781 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:51.676866 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:52.177308 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:52.177417 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:52.477265 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:52.477282 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:34:52.477285 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:34:52.477289 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:34:52.477297 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:52.477301 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:52.477303 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:52.477341 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:52.572641 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:52.572755 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:52.601813 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:52.601912 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:52.705652 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:52.705771 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:52.769836 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:52.770400 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:52.853558 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:52.853670 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:52.942261 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:53.102346 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:53.102422 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:53.336611 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.336641 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.336686 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.337414 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.337480 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.337568 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.337607 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.350876 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.350900 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.358254 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.363287 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.384858 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:53.388367 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.388390 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.406203 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.406227 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.414474 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:53.414477 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.414511 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.414532 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.414588 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:53.459555 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:53.459632 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:53.477374 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:53.477394 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:53.477399 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:53.477401 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:53.477442 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:53.543677 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:53.543738 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:53.969383 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:53.969593 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:54.437186 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:54.437204 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:54.437743 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:54.437802 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:54.437829 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:54.437849 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:54.437869 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:54.437889 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:54.438047 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:54.438086 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:54.438114 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:54.438139 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:54.438164 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:54.438186 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:54.438207 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:54.438229 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:54.438251 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:54.438273 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:54.477461 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:54.477478 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:54.477481 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:54.477483 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:54.477515 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:54.769928 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:54.770478 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:54.942348 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:55.077665 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:55.077838 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:55.336694 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.336721 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.336747 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.337494 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.337567 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.337632 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.337676 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.350977 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.351037 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.358352 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.363385 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.384938 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:55.388468 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.388528 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.406303 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.406331 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.414583 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:55.414587 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.414653 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.414683 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.414731 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:55.473179 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:55.473243 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:55.477563 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:55.477585 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:55.477589 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:55.477592 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:55.477631 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:55.537839 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:55.537857 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:55.538331 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:55.538372 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:55.538428 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:55.538557 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:55.538592 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:55.538628 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:55.538666 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:55.538688 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:55.538707 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:55.538746 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:55.538773 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:55.538796 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:55.538821 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:55.538842 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:55.538864 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:55.538887 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:55.844293 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:55.844341 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:56.002888 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:56.002990 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:56.206135 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:56.206217 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:56.354079 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:56.354135 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:56.359968 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:56.360036 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:56.477674 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:56.477695 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:56.477700 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:56.477702 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:56.477742 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:56.770024 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:56.770567 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:56.942441 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:57.306662 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:57.306778 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:57.336779 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.336797 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.336823 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.337577 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.337659 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.337685 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.337744 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.351074 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.351114 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.358450 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.363484 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.385032 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:57.388549 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.388606 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.406404 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.406431 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.414691 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:57.414695 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.414742 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.414760 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.414815 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:57.454631 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:57.454827 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:57.477788 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:57.477806 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:34:57.477812 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:34:57.477817 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:34:57.477865 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:57.477860 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.33 < max 0.5 = yes 2014-05-19 18:34:57.477877 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:34:57.477883 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 140.202 < min (86400 seconds) 2014-05-19 18:34:57.477919 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:34:57.477925 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:57.477930 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:57.773687 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:57.773777 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:58.478014 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:58.478035 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:58.478040 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:58.478042 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:58.478082 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:58.660512 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:58.660615 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:58.770122 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:34:58.770645 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:58.942531 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:34:59.038427 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:59.038446 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:34:59.039036 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:59.039088 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:34:59.039161 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:59.039211 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:59.039242 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:59.039261 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:59.039288 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:34:59.039313 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:59.039341 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:59.039365 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:59.039392 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:59.039428 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:59.039449 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:34:59.039471 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:34:59.039490 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:59.039514 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:34:59.336856 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.336875 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.336901 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.337662 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.337740 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.337771 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.337822 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.344797 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:59.344926 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:34:59.351171 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.351191 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.358546 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.363568 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.385129 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:34:59.388632 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.388688 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.406506 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.406531 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.414797 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:34:59.414801 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.414820 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.414837 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.414886 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:34:59.478129 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:34:59.478150 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:34:59.478154 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:34:59.478156 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:34:59.478196 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:34:59.777981 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:59.778060 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:34:59.869928 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:34:59.870036 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:00.445294 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:00.445346 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:00.478243 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:00.478264 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:00.478269 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:00.478271 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:00.478311 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:00.703169 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:00.703229 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:00.770216 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:00.770725 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:00.942626 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:00.955235 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:00.955340 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:01.274191 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:01.274255 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:01.336938 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.336956 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.337006 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.337743 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.337823 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.337876 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.337903 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.351269 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.351294 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.358643 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.363651 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.385223 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:01.388712 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.388768 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.406606 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.406633 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.414895 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:01.414899 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.414934 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.414957 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.414986 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:01.478359 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:01.478379 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:01.478384 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:01.478386 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:01.478426 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:02.161039 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:02.161113 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:02.374686 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:02.374756 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:02.478471 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:02.478490 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:35:02.478495 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:35:02.478501 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:35:02.478549 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:02.478544 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.3 < max 0.5 = yes 2014-05-19 18:35:02.478562 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:02.478568 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 145.203 < min (86400 seconds) 2014-05-19 18:35:02.478593 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:02.478598 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:02.478602 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:02.607081 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:02.607202 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:02.770316 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:02.770799 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:02.942712 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:03.261428 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:03.261533 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:03.337030 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.337055 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.337082 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.337826 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.337905 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.337955 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.337983 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.351368 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.351392 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.358741 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.363750 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.385318 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:03.388796 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.388845 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.406708 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.406734 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.415002 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:03.415005 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.415039 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.415058 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.415081 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:03.478666 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:03.478686 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:03.478689 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:03.478691 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:03.478721 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:03.945816 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:03.945879 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:04.339076 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:04.339094 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:04.339587 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:04.339666 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:04.339721 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:04.339767 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:04.339796 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:04.339819 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:04.339846 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:04.339871 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:04.339896 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:04.339921 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:04.339949 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:04.339969 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:04.339989 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:04.340049 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:04.340085 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:04.340110 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:04.478779 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:04.478800 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:04.478805 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:04.478808 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:04.478847 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:04.570603 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:04.570722 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:04.770401 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:04.770884 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:04.942800 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:05.337109 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.337153 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.337168 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.337907 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.337988 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.338033 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.338061 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.351464 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.351489 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.358840 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.363846 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.385414 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:05.388883 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.388923 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.406809 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.406836 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.415108 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:05.415111 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.415148 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.415168 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.415216 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:05.478896 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:05.478945 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.28 < max 0.5 = yes 2014-05-19 18:35:05.478958 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:05.478964 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:05.478961 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 148.203 < min (86400 seconds) 2014-05-19 18:35:05.478980 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:05.478983 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:05.478985 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:05.678277 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:05.678357 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:06.255711 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:06.255778 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:06.479067 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:06.479115 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.28 < max 0.5 = yes 2014-05-19 18:35:06.479128 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:06.479135 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:06.479131 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 149.203 < min (86400 seconds) 2014-05-19 18:35:06.479153 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:06.479156 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:06.479158 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:06.603479 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:06.603574 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:06.770494 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:06.770963 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:06.942893 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:07.337198 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.337240 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.337255 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.337997 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.338078 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.338105 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.338167 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.351564 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.351619 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.358937 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.363942 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.385503 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:07.388963 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.389001 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.406910 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.406934 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.415216 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:07.415219 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.415255 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.415277 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.415298 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:07.479244 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:07.479261 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:35:07.479265 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:35:07.479269 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:35:07.479305 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.28 < max 0.5 = yes 2014-05-19 18:35:07.479317 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:07.479320 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:07.479320 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 150.203 < min (86400 seconds) 2014-05-19 18:35:07.479339 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:07.479342 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:07.479343 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:07.675212 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:07.675319 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:08.479426 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:08.479474 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.25 < max 0.5 = yes 2014-05-19 18:35:08.479486 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:08.479494 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:08.479489 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 151.204 < min (86400 seconds) 2014-05-19 18:35:08.479508 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:08.479511 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:08.479513 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:08.507521 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:08.507582 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:08.561720 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:08.561784 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:08.770592 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:08.771042 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:08.942980 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:09.246247 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:09.246305 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:09.271087 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:09.271187 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:09.337293 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.337314 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.337329 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.338093 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.338162 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.338186 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.338237 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.351662 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.351702 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.359033 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.364038 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.385598 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:09.389043 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.389079 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.407012 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.407071 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.415324 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:09.415327 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.415362 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.415381 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.415401 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:09.479601 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:09.479622 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:09.479628 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:09.479630 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:09.479669 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:09.608041 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:09.608110 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:09.639635 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:09.639654 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:09.640166 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:09.640235 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:09.640292 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:09.640327 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:09.640358 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:09.640393 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:09.640444 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:09.640486 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:09.640515 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:09.640538 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:09.640564 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:09.640605 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:09.640629 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:09.640715 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:09.640757 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:09.640780 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:10.378758 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:10.378831 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:10.479711 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:10.479732 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:10.479737 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:10.479739 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:10.479779 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:10.770690 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:10.771120 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:10.943072 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:11.303841 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:11.303922 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:11.337390 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.337415 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.337467 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.338174 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.338248 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.338281 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.338298 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.351756 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.351781 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.359130 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.364118 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.385693 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:11.389124 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.389158 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.407112 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.407148 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.415432 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:11.415434 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.415473 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.415492 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.415547 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:11.479828 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:11.479848 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:11.479853 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:11.479855 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:11.479895 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:11.556311 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:11.556401 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:11.571632 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:11.571723 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:12.479939 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:12.479956 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:35:12.479960 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:35:12.479964 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:35:12.479971 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:12.479975 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:12.479977 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:12.480015 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:12.540253 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:12.540271 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:12.540804 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:12.540849 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:12.540895 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:12.540934 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:12.540969 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:12.540992 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:12.541022 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:12.541042 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:12.541067 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:12.541087 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:12.541134 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:12.541187 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:12.541216 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:12.541238 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:12.541264 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:12.541292 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:12.770787 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:12.771199 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:12.943159 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:13.337485 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.337545 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.337566 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.338249 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.338342 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.338368 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.338426 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.351855 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.351879 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.359184 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.364214 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.385786 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:13.389207 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.389238 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.407211 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.407228 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.415537 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:13.415541 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.415576 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.415597 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.415628 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:13.480065 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:13.480087 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:13.480092 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:13.480094 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:13.480133 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:13.575780 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:13.575851 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:14.456852 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:14.456970 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:14.461977 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:14.462046 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:14.480181 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:14.480202 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:14.480206 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:14.480209 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:14.480249 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:14.546701 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:14.546749 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:14.676348 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:14.676431 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:14.770885 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:14.771277 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:14.943254 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:15.079067 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:15.079148 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:15.337568 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.337624 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.337659 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.338327 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.338424 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.338446 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.338500 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.351953 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.351977 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.359276 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.364310 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.385884 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:15.389288 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.389318 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.407311 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.407336 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.415642 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:15.415645 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.415677 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.415699 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.415717 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:15.480297 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:15.480318 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:15.480322 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:15.480324 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:15.480347 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:15.508538 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:15.508668 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:15.562497 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:15.562547 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:15.672276 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:15.672401 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:16.004191 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:16.004246 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:16.480397 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:16.480418 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:16.480423 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:16.480426 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:16.480465 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:16.770982 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:16.771350 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:16.943349 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:17.337645 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.337709 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.337729 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.338420 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.338506 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.338525 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.338577 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.352051 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.352074 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.359327 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.364390 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.379480 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:17.379584 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:17.385978 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:17.389374 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.389397 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.407411 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.407438 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.415751 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:17.415753 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.415791 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.415814 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.415833 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:17.447162 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:17.447241 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:17.480499 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:17.480516 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:35:17.480520 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:35:17.480523 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:35:17.480558 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.23 < max 0.5 = yes 2014-05-19 18:35:17.480570 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:17.480603 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:17.480573 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 160.205 < min (86400 seconds) 2014-05-19 18:35:17.480618 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:17.480622 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:17.480623 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:17.840861 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:17.840879 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:17.841451 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:17.841520 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:17.841548 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:17.841607 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:17.841648 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:17.841673 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:17.841698 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:17.841717 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:17.841784 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:17.841814 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:17.841847 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:17.841867 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:17.841888 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:17.841907 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:17.841929 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:17.841954 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:18.304573 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:18.304664 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:18.480697 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:18.480745 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.21 < max 0.5 = yes 2014-05-19 18:35:18.480758 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:18.480766 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:18.480761 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 161.205 < min (86400 seconds) 2014-05-19 18:35:18.480780 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:18.480782 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:18.480784 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:18.771080 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:18.771408 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:18.943423 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:19.337734 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.337778 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.337804 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.338502 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.338589 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.338607 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.338635 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.352150 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.352176 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.359419 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.364470 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.386073 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:19.389477 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.389502 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.407510 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.407531 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.415858 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:19.415861 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.415897 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.415921 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.415940 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:19.480862 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:19.480910 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.21 < max 0.5 = yes 2014-05-19 18:35:19.480923 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:19.480929 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:19.480927 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 162.205 < min (86400 seconds) 2014-05-19 18:35:19.480945 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:19.480948 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:19.480950 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:19.608991 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:19.609102 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:19.757385 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:19.757450 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:20.481034 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:20.481055 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:20.481059 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:20.481062 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:20.481103 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:20.576872 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:20.576967 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:20.771179 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:20.771471 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:20.943515 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:20.972793 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:20.972887 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:21.337823 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.337851 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.337873 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.338582 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.338671 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.338698 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.338715 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.352240 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.352266 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.359517 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.364551 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.386168 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:21.389578 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.389603 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.407609 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.407635 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.415964 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:21.415966 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.415999 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.416028 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.416050 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:21.462749 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:21.462860 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:21.479820 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:21.479960 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:21.481149 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:21.481197 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.21 < max 0.5 = yes 2014-05-19 18:35:21.481210 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:21.481206 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:21.481213 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 164.205 < min (86400 seconds) 2014-05-19 18:35:21.481232 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:21.481235 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:21.481236 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:22.404908 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:22.404998 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:22.481320 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:22.481337 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:35:22.481341 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:35:22.481345 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:35:22.481352 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:22.481356 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:22.481358 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:22.481371 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:22.747560 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:22.747673 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:22.771277 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:22.771548 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:22.943599 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:23.141504 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:23.141522 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:23.142054 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:23.142097 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:23.142124 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:23.142166 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:23.142188 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:23.142214 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:23.142247 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:23.142322 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:23.142360 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:23.142398 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:23.142438 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:23.142460 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:23.142492 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:23.142514 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:23.142536 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:23.142558 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:23.337914 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.337938 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.337962 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.338664 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.338755 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.338776 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.338802 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.352337 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.352361 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.359609 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.364634 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.386263 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:23.389678 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.389706 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.407710 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.407737 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.416070 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:23.416073 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.416104 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.416127 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.416141 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:23.481445 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:23.481494 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.2 < max 0.5 = yes 2014-05-19 18:35:23.481507 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:23.481513 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:23.481510 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 166.206 < min (86400 seconds) 2014-05-19 18:35:23.481528 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:23.481531 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:23.481533 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:24.481621 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:24.481642 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:24.481646 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:24.481648 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:24.481690 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:24.677632 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:24.677691 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:24.771377 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:24.771618 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:24.909486 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:24.909559 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:24.943696 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:25.057912 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:25.058044 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:25.338013 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.338038 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.338057 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.338745 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.338839 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.338866 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.338888 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.352420 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.352438 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.359700 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.364716 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.386358 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:25.389779 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.389804 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.407809 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.407833 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.416176 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:25.416179 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.416215 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.416236 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.416287 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:25.442129 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:25.442145 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:25.442676 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:25.442774 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:25.442829 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:25.442873 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:25.442933 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:25.442999 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:25.443049 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:25.443095 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:25.443136 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:25.443177 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:25.443205 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:25.443225 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:25.443248 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:25.443271 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:25.443293 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:25.443315 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:25.481719 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:25.481741 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:25.481746 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:25.481748 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:25.481777 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:25.563079 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:25.563153 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:25.942772 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:25.942790 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:25.943318 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:25.943369 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:25.943397 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:25.943458 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:25.943497 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:25.943519 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:25.943541 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:25.943576 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:25.943600 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:25.943637 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:25.943686 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:25.943716 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:25.943742 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:25.943765 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:25.943787 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:25.943810 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:26.273373 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:26.273491 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:26.481831 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:26.481852 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:26.481856 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:26.481859 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:26.481898 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:26.771461 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:26.771696 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:26.780081 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:26.780188 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:26.943791 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:27.209957 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:27.210077 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:27.338111 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.338131 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.338150 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.338834 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.338921 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.338947 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.338969 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.352501 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.352520 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.358458 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:27.358516 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:27.359769 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.364798 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.386454 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:27.389879 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.389904 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.407881 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.407910 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.416283 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:27.416286 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.416322 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.416342 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.416369 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:27.481944 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:27.481961 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:35:27.481965 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:35:27.481969 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:35:27.481972 7fd04f907700 10 osd.0 21 send_pg_stats - 0 pgs updated 2014-05-19 18:35:27.482004 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:27.482008 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:27.482010 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:27.482044 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:27.643391 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:27.643409 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:27.643929 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:27.643973 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:27.644086 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:27.644117 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:27.644140 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:27.644160 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:27.644190 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:27.644228 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:27.644275 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:27.644332 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:27.644362 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:27.644386 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:27.644410 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:27.644429 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:27.644451 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:27.644477 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:27.705172 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:27.705254 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:27.710423 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:27.710485 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:27.858981 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:27.859043 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:28.048062 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:28.048116 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:28.144007 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:28.144026 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:28.144569 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:28.144657 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:28.144697 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:28.144749 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:28.144795 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:28.144834 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:28.144879 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:28.144962 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:28.144990 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:28.145010 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:28.145034 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:28.145057 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:28.145090 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:28.145127 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:28.145151 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:28.145171 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:28.364940 7fd04aecb700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:28.364945 7fd04aecb700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:28.364947 7fd04aecb700 20 osd.0 21 _dispatch 0x8389180 pg_stats_ack(0 pgs tid 7) v1 2014-05-19 18:35:28.364953 7fd04aecb700 10 osd.0 21 handle_pg_stats_ack 2014-05-19 18:35:28.364957 7fd04aecb700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:28.364959 7fd04aecb700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:28.482091 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:28.482112 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:28.482117 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:28.482119 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:28.482160 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:28.573934 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:28.574012 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:28.771545 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:28.771763 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:28.943883 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:29.338167 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.338196 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.338226 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.338903 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.339003 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.339025 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.339050 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.352594 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.352621 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.359840 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.364880 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.386550 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:29.389979 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.390003 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.407978 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.408002 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.410973 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:29.411030 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:29.416379 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:29.416381 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.416417 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.416438 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.416461 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:29.482205 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:29.482225 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:29.482230 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:29.482232 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:29.482274 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:29.559566 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:29.559666 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:29.911487 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:29.911582 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:29.978302 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:29.978381 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:30.060156 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:30.060221 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:30.348538 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:30.348696 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:30.482315 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:30.482336 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:30.482340 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:30.482343 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:30.482366 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:30.771626 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:30.771829 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:30.849034 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:30.849111 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:30.863394 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:30.863448 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:30.943971 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:31.338262 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.338284 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.338299 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.338968 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.339083 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.339106 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.339126 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.352676 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.352704 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.359931 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.364984 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.386645 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:31.390079 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.390107 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.408078 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.408100 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.416473 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:31.416474 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.416510 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.416526 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.416550 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:31.482430 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:31.482482 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.18 < max 0.5 = yes 2014-05-19 18:35:31.482496 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:31.482499 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:31.482499 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 174.207 < min (86400 seconds) 2014-05-19 18:35:31.482518 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:31.482521 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:31.482523 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:31.644635 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:31.644653 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:31.645186 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:31.645241 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:31.645280 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:31.645301 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:31.645344 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:31.645370 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:31.645413 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:31.645437 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:31.645460 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:31.645483 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:31.645546 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:31.645577 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:31.645600 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:31.645619 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:31.645638 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:31.645662 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:32.080377 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:32.080442 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:32.278740 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:32.278834 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:32.482607 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:32.482625 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:35:32.482631 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:35:32.482637 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:35:32.482686 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:32.482680 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.18 < max 0.5 = yes 2014-05-19 18:35:32.482697 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:32.482702 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 175.207 < min (86400 seconds) 2014-05-19 18:35:32.482737 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:32.482742 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:32.482745 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:32.549540 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:32.549708 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:32.771722 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:32.771899 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:32.779365 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:32.779501 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:32.944056 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:33.005415 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:33.005489 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:33.050043 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:33.050150 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:33.163860 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:33.163997 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:33.338359 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.338398 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.338448 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.339036 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.339165 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.339183 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.339203 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.345286 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:33.345304 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:33.345916 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:33.345970 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:33.345999 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:33.346042 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:33.346063 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:33.346089 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:33.346111 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:33.346155 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:33.346247 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:33.346284 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:33.346315 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:33.346338 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:33.346366 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:33.346388 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:33.346410 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:33.346435 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:33.352750 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.352781 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.360030 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.365067 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.386742 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:33.390181 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.390205 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.408177 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.408204 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.411942 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:33.411999 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:33.416586 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.416572 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:33.416612 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.416624 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.416650 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:33.482835 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:33.482858 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:33.482865 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:33.482869 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:33.482896 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:33.560700 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:33.560770 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:33.664337 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:33.664424 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:34.380692 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:34.380749 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:34.474591 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:34.474753 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:34.479874 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:34.479945 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:34.482959 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:34.483009 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:34.483007 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.17 < max 0.5 = yes 2014-05-19 18:35:34.483020 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:34.483023 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 177.207 < min (86400 seconds) 2014-05-19 18:35:34.483043 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:34.483046 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:34.483048 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:34.771821 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:34.771974 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:34.944150 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:34.980478 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:34.980547 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:35.112445 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:35.112510 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:35.261267 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:35.261331 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:35.305843 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:35.305892 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:35.338456 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.338476 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.338537 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.339105 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.339241 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.339262 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.339278 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.352841 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.352859 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.360126 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.364755 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:35.364845 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:35.365137 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.386838 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:35.390281 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.390309 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.408263 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.408282 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.416673 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.416673 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:35.416705 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.416729 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.416780 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:35.483135 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:35.483185 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:35.483183 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.17 < max 0.5 = yes 2014-05-19 18:35:35.483196 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:35.483199 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 178.207 < min (86400 seconds) 2014-05-19 18:35:35.483218 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:35.483221 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:35.483223 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:35.865297 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:35.865382 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:36.245986 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:36.246005 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:36.246463 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:36.246501 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:36.246546 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:36.246576 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:36.246597 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:36.246616 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:36.246662 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:36.246723 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:36.246767 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:36.246791 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:36.246813 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:36.246838 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:36.246912 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:36.246940 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:36.246969 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:36.246990 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:36.483295 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:36.483319 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:36.483325 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:36.483330 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:36.483356 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:36.550514 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:36.550614 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:36.771917 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:36.772042 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:36.944236 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:37.338526 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.338552 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.338616 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.339175 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.339311 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.339339 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.339357 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.352941 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.353000 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.360181 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.365220 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.375184 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:37.375266 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:37.386919 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:37.390352 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.390389 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.408333 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.408358 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.416750 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.416763 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:37.416802 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.416811 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.416854 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:37.483401 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:37.483418 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:35:37.483422 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:35:37.483426 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:35:37.483433 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:37.483438 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:37.483440 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:37.483478 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:38.012924 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:38.012976 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:38.161826 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:38.161876 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:38.251162 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:38.251227 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:38.481122 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:38.481206 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:38.483520 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:38.483540 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:38.483545 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:38.483547 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:38.483588 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:38.772014 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:38.772118 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:38.944308 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:39.075679 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:39.075784 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:39.146550 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:39.146569 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:39.147092 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:39.147141 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:39.147203 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:39.147284 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:39.147328 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:39.147367 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:39.147432 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:39.147484 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:39.147530 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:39.147570 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:39.147618 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:39.147660 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:39.147706 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:39.147749 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:39.147793 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:39.147834 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:39.338622 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.338645 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.338697 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.339239 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.339383 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.339415 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.339438 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.353042 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.353078 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.360274 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.365300 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.365663 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:39.365728 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:39.387016 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:39.390446 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.390468 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.408412 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.408441 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.416841 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:39.416846 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.416878 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.416907 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.416931 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:39.483621 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:39.483642 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:39.483646 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:39.483649 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:39.483690 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:40.181830 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:40.181922 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:40.281019 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:40.281082 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:40.483716 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:40.483764 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.15 < max 0.5 = yes 2014-05-19 18:35:40.483777 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:40.483773 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:40.483780 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 183.208 < min (86400 seconds) 2014-05-19 18:35:40.483799 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:40.483802 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:40.483803 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:40.772111 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:40.772196 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:40.913479 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:40.913575 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:40.944387 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:41.062408 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:41.062531 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:41.065968 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:41.066072 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:41.151687 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:41.151795 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:41.206234 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:41.206302 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:41.338718 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.338743 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.338774 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.339304 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.339461 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.339490 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.339509 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.353123 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.353156 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.360353 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.365398 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.387111 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:41.390546 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.390572 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.408498 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.408520 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.416952 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.416953 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:41.416984 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.417018 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.417043 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:41.483892 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:41.483912 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:41.483917 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:41.483919 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:41.483960 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:42.484004 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:42.484021 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:35:42.484025 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:35:42.484028 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:35:42.484036 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:42.484040 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:42.484042 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:42.484081 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:42.772201 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:42.772257 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:42.944460 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:43.082626 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:43.082721 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:43.338817 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.338841 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.338861 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.339367 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.339543 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.339578 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.339599 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.353221 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.353246 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.360434 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.365495 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.387197 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:43.390645 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.390670 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.408598 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.408623 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.417058 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:43.417060 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.417095 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.417109 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.417125 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:43.484128 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:43.484149 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:43.484154 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:43.484156 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:43.484197 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:43.776231 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:43.776350 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:43.966388 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:43.966456 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:44.052144 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:44.052205 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:44.447161 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:44.447179 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:44.447773 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:44.447828 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:44.447882 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:44.447908 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:44.447943 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:44.447967 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:44.447997 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:44.448021 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:44.448042 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:44.448107 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:44.448172 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:44.448218 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:44.448245 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:44.448268 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:44.448289 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:44.448311 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:44.484245 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:44.484266 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:44.484270 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:44.484273 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:44.484302 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:44.772299 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:44.772328 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:44.944535 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:45.338879 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.338933 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.338949 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.339434 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.339617 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.339675 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.339706 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.353324 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.353378 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.360483 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.365591 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.387276 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:45.390744 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.390768 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.408686 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.408705 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.417163 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:45.417168 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.417202 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.417225 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.417288 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:45.476749 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:45.476872 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:45.484335 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:45.484352 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:45.484356 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:45.484358 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:45.484358 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:45.983134 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:45.983218 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:46.147861 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:46.147879 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:46.148401 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:46.148479 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:46.148527 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:46.148552 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:46.148607 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:46.148644 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:46.148704 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:46.148810 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:46.148838 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:46.148873 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:46.148899 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:46.148921 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:46.148945 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:46.148967 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:46.148989 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:46.149014 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:46.181329 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:46.181473 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:46.213988 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:46.214056 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:46.362980 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:46.363061 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:46.484431 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:46.484453 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:46.484457 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:46.484460 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:46.484501 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:46.772383 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:46.772402 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:46.866683 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:46.866832 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:46.944614 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:47.106382 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:47.106480 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:47.338955 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.339004 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.339021 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.339504 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.339686 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.339744 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.339784 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.353424 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.353464 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.360562 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.365678 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.387373 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:47.390845 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.390871 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.408787 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.408812 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.417272 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:47.417275 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.417310 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.417329 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.417359 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:47.484530 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:47.484547 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:35:47.484551 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:35:47.484555 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:35:47.484563 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:47.484567 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:47.484569 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:47.484607 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:47.777469 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:47.777515 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:47.914413 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:47.914541 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:48.063499 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:48.063578 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:48.277928 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:48.277988 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:48.484635 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:48.484684 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.13 < max 0.5 = yes 2014-05-19 18:35:48.484697 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:48.484704 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:48.484700 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 191.209 < min (86400 seconds) 2014-05-19 18:35:48.484719 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:48.484722 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:48.484724 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:48.772466 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:48.772481 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:48.944691 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:49.339006 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.339081 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.339111 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.339574 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.339754 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.339812 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.339853 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.352568 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:49.352676 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:49.353504 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.353567 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.360649 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.365775 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.387459 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:49.390947 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.390971 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.408888 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.408911 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.417378 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:49.417381 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.417465 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.417487 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.417543 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:49.484796 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:49.484817 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:49.484822 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:49.484824 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:49.484865 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:49.648462 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:49.648480 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:49.648951 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:49.649036 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:49.649107 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:49.649154 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:49.649195 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:49.649307 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:49.649362 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:49.649408 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:49.649449 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:49.649494 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:49.649543 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:49.649588 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:49.649634 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:49.649678 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:49.649720 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:49.649763 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:49.978566 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:49.978726 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:50.484891 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:50.484939 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.13 < max 0.5 = yes 2014-05-19 18:35:50.484953 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:50.484948 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:50.484956 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 193.209 < min (86400 seconds) 2014-05-19 18:35:50.484975 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:50.484977 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:50.484979 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:50.772553 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:50.772557 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:50.881657 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:50.881751 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:50.944782 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:51.053049 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:51.053139 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:51.283812 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:51.283893 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:51.339100 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.339175 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.339222 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.339656 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.339835 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.339884 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.339936 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.353601 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.353653 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.360736 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.365872 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.387540 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:51.391047 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.391075 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.408989 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.409015 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.415021 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:51.415091 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:51.417496 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:51.417498 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.417543 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.417570 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.417627 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:51.485044 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:51.485065 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:51.485071 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:51.485074 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:51.485102 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:51.563993 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:51.564115 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:51.806719 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:51.806788 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:52.167019 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:52.167099 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:52.485149 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:52.485165 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:35:52.485169 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:35:52.485173 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:35:52.485181 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:52.485184 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:52.485187 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:52.485200 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:52.582073 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:52.582160 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:52.772667 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:52.772671 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:52.944867 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:52.984477 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:52.984612 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:53.149040 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:53.149059 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:53.149570 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:53.149630 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:53.149674 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:53.149697 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:53.149719 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:53.149742 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:53.149762 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:53.149854 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:53.149929 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:53.149960 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:53.149992 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:53.150029 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:53.150058 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:53.150080 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:53.150104 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:53.150129 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:53.339175 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.339264 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.339321 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.339726 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.339915 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.339983 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.340039 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.353694 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.353731 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.360832 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.365970 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.387626 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:53.391148 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.391173 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.409072 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.409096 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.417602 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:53.417605 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.417641 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.417663 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.417704 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:53.479093 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:53.479202 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:53.485259 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:53.485309 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.12 < max 0.5 = yes 2014-05-19 18:35:53.485323 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:53.485328 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:53.485326 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 196.209 < min (86400 seconds) 2014-05-19 18:35:53.485344 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:53.485347 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:53.485349 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:53.507085 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:53.507156 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:53.867486 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:53.867569 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:54.485433 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:54.485453 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:54.485458 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:54.485460 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:54.485501 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:54.553564 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:54.553645 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:54.772753 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:54.772755 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:54.882369 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:54.882469 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:54.915495 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:54.915557 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:54.944960 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:55.064693 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:55.064818 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:55.339256 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.339342 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.339405 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.339795 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.339996 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.340071 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.340104 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.353790 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.353807 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.360924 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.366067 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.382921 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:55.383000 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:55.387723 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:55.391252 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.391275 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.409172 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.409196 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.417710 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:55.417713 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.417749 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.417768 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.417783 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:55.485546 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:55.485567 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:55.485572 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:55.485574 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:55.485615 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:55.807462 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:55.807554 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:56.307937 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:56.307994 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:56.483266 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:56.483339 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:56.485097 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:56.485195 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:56.485637 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:56.485652 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:56.485655 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:56.485657 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:56.485689 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:56.772846 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:56.772850 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:56.945054 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:56.979663 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:56.979798 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:57.249612 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:57.249631 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:57.250164 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:57.250199 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:57.250275 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:57.250340 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:57.250398 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:57.250442 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:57.250474 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:57.250497 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:57.250520 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:57.250542 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:57.250569 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:57.250591 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:57.250616 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:57.250641 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:57.250665 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:57.250690 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:57.339333 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.339420 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.339481 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.339880 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.340062 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.340159 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.340178 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.353889 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.353913 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.361014 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.366164 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.367885 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:57.367976 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:57.387820 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:57.391353 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.391379 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.408369 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:57.408465 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:57.409252 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.409274 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.417816 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:57.417817 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.417853 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.417873 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.417921 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:57.485744 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:57.485761 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:35:57.485765 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:35:57.485769 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:35:57.485777 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:57.485781 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:57.485783 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:57.485821 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:58.053995 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:58.054046 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:58.485865 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:58.485885 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:35:58.485890 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:58.485892 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:58.485933 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:58.772901 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:58.772931 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:35:58.945139 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:35:58.950165 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:58.950184 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:35:58.950654 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:58.950719 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:58.950756 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:58.950785 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:35:58.950822 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:58.950851 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:58.950874 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:58.950981 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:58.951013 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:58.951033 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:58.951061 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:35:58.951083 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:58.951111 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:58.951134 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:58.951155 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:35:58.951177 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:35:59.015978 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:59.016083 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:35:59.165312 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:59.165366 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:35:59.339411 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.339494 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.339560 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.339961 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.340140 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.340242 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.340266 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.353986 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.354012 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.361111 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.366261 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.387916 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:35:59.391451 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.391476 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.409350 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.409376 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.417919 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:35:59.417922 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.417961 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.417985 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.418014 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:35:59.485980 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:35:59.486028 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.11 < max 0.5 = yes 2014-05-19 18:35:59.486042 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:35:59.486049 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:35:59.486045 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 202.21 < min (86400 seconds) 2014-05-19 18:35:59.486063 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:35:59.486067 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:35:59.486068 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:35:59.983653 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:59.983743 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:35:59.985700 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:35:59.985752 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:00.486151 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:00.486174 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:00.486181 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:00.486185 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:00.486212 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:00.716409 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:00.716495 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:00.772977 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:00.773009 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:00.865753 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:00.865883 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:00.868164 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:00.868254 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:00.908669 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:00.908732 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:00.945235 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:01.080311 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:01.080428 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:01.339504 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.339579 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.339638 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.340041 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.340216 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.340321 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.340342 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.354084 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.354108 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.361208 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.366358 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.388013 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:01.391550 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.391574 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.409450 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.409474 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.418024 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:01.418030 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.418053 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.418067 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.418092 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:01.486273 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:01.486325 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:01.486321 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.11 < max 0.5 = yes 2014-05-19 18:36:01.486335 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:01.486338 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 204.21 < min (86400 seconds) 2014-05-19 18:36:01.486356 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:01.486360 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:01.486361 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:02.154578 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:02.154623 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:02.486443 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:02.486461 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:36:02.486467 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:36:02.486473 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:36:02.486522 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:02.486515 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.11 < max 0.5 = yes 2014-05-19 18:36:02.486533 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:02.486538 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 205.211 < min (86400 seconds) 2014-05-19 18:36:02.486575 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:02.486581 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:02.486584 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:02.773043 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:02.773122 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:02.780760 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:02.780852 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:02.945321 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:03.339589 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.339656 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.339717 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.340122 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.340302 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.340404 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.340432 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.354183 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.354206 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.361300 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.366456 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.388107 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:03.391650 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.391677 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.409550 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.409573 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.418109 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:03.418113 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.418141 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.418164 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.418184 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:03.486672 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:03.486692 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:03.486697 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:03.486699 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:03.486722 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:03.650744 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:03.650763 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:03.651218 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:03.651264 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:03.651313 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:03.651378 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:03.651407 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:03.651427 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:03.651483 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:03.651599 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:03.651651 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:03.651677 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:03.651705 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:03.651728 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:03.651754 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:03.651776 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:03.651797 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:03.651823 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:03.855006 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:03.855057 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:04.086456 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:04.086529 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:04.486785 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:04.486834 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.1 < max 0.5 = yes 2014-05-19 18:36:04.486847 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:04.486854 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:04.486850 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 207.211 < min (86400 seconds) 2014-05-19 18:36:04.486870 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:04.486873 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:04.486875 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:04.773100 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:04.773206 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:04.945405 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:04.968702 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:04.968795 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:05.283930 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:05.284029 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:05.339683 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.339730 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.339794 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.340203 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.340384 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.340488 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.340509 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.354282 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.354307 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.361381 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.366553 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.388209 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:05.391749 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.391775 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.409649 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.409676 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.417035 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:05.417104 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:05.418197 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:05.418198 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.418237 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.418257 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.418286 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:05.486962 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:05.486983 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:05.486988 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:05.486990 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:05.487019 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:05.566346 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:05.566472 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:05.787066 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:05.787147 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:06.208926 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:06.209001 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:06.487077 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:06.487097 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:06.487102 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:06.487104 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:06.487145 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:06.669216 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:06.669295 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:06.773176 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:06.773287 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:06.945502 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:06.984402 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:06.984507 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:07.339778 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.339802 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.339880 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.340288 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.340466 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.340570 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.340620 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.354380 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.354406 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.361458 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.366650 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.388290 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:07.391850 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.391878 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.409750 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.409774 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.418286 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:07.418288 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.418326 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.418349 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.418370 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:07.481277 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:07.481396 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:07.487171 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:07.487188 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:36:07.487191 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:36:07.487194 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:36:07.487202 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:07.487205 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:07.487207 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:07.487208 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:07.909289 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:07.909342 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:08.487294 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:08.487342 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.09 < max 0.5 = yes 2014-05-19 18:36:08.487356 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:08.487352 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:08.487359 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 211.211 < min (86400 seconds) 2014-05-19 18:36:08.487378 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:08.487381 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:08.487382 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:08.555728 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:08.555774 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:08.773257 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:08.773368 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:08.945588 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:09.339833 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.339878 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.339951 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.340368 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.340504 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.340670 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.340692 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.354479 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.354502 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.361499 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.366747 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.388373 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:09.391950 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.391976 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.409849 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.409872 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.418394 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:09.418395 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.418429 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.418450 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.418508 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:09.487470 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:09.487491 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:09.487495 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:09.487498 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:09.487539 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:09.551314 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:09.551332 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:09.551781 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:09.551826 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:09.551878 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:09.551926 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:09.552000 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:09.552097 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:09.552164 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:09.552207 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:09.552250 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:09.552292 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:09.552339 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:09.552383 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:09.552429 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:09.552473 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:09.552512 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:09.552556 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:10.487485 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:10.487535 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:10.487574 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:10.487596 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:10.487602 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:10.487605 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:10.487610 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:10.651874 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:10.651893 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:10.652351 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:10.652409 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:10.652447 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:10.652472 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:10.652498 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:10.652552 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:10.652632 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:10.652737 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:10.652785 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:10.652812 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:10.652841 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:10.652866 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:10.652893 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:10.652918 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:10.652940 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:10.652964 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:10.773351 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:10.773454 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:10.945680 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:11.084694 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:11.084780 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:11.317576 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:11.317632 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:11.339929 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.339955 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.340031 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.340440 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.340620 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.340742 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.340761 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.354576 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.354603 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.361593 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.366846 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.369584 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:11.369649 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:11.388454 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:11.392042 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.392061 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.409949 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.409975 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.418497 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:11.418504 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.418525 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.418545 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.418616 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:11.467096 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:11.467304 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:11.487695 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:11.487717 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:11.487722 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:11.487724 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:11.487764 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:12.009679 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:12.009756 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:12.418249 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:12.418328 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:12.487808 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:12.487826 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:36:12.487831 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:36:12.487837 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:36:12.487849 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:12.487854 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:12.487858 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:12.487886 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:12.567562 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:12.567669 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:12.773447 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:12.773535 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:12.785083 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:12.785168 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:12.945767 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:12.952430 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:12.952449 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:12.952926 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:12.952980 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:12.953026 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:12.953193 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:12.953219 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:12.953243 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:12.953262 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:12.953352 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:12.953399 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:12.953424 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:12.953450 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:12.953473 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:12.953499 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:12.953525 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:12.953548 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:12.953570 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:13.340012 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.340030 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.340102 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.340510 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.340695 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.340818 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.340847 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.354675 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.354699 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.361692 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.366944 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.381878 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:13.381944 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:13.388536 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:13.392124 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.392147 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.410048 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.410073 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.418594 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:13.418601 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.418623 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.418641 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.418681 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:13.487947 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:13.487968 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:13.487973 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:13.487975 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:13.488016 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:13.710029 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:13.710097 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:14.456368 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:14.456426 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:14.482402 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:14.482463 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:14.488046 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:14.488067 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:14.488072 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:14.488074 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:14.488115 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:14.718705 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:14.718801 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:14.773537 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:14.773617 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:14.868095 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:14.868200 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:14.945862 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:15.253018 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:15.253037 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:15.253494 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:15.253561 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:15.253647 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:15.253686 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:15.253746 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:15.253800 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:15.253866 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:15.253888 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:15.253945 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:15.253974 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:15.254003 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:15.254026 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:15.254050 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:15.254073 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:15.254095 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:15.254117 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:15.340095 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.340119 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.340206 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.340588 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.340767 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.340891 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.340944 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.354775 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.354831 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.361789 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.367041 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.388617 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:15.392207 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.392226 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.410148 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.410171 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.418689 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:15.418691 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.418730 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.418750 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.418767 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:15.488148 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:15.488168 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:15.488173 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:15.488175 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:15.488217 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:15.556908 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:15.557031 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:16.387936 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:16.388029 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:16.488242 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:16.488301 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:16.488298 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.09 < max 0.5 = yes 2014-05-19 18:36:16.488330 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:16.488336 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 219.212 < min (86400 seconds) 2014-05-19 18:36:16.488360 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:16.488366 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:16.488370 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:16.773630 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:16.773703 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:16.783027 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:16.783103 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:16.885515 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:16.885622 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:16.945958 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:17.019255 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:17.019349 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:17.168706 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:17.168796 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:17.269757 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:17.269859 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:17.340173 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.340194 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.340286 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.340670 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.340837 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.340962 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.341028 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.354873 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.354916 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.361887 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.367139 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.388689 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:17.392288 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.392309 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.410247 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.410273 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.418798 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:17.418803 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.418823 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.418848 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.418894 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:17.488368 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:17.488417 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:17.488435 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:17.488447 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:36:17.488450 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:36:17.488455 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:36:17.488489 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:17.488502 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.09 < max 0.5 = yes 2014-05-19 18:36:17.488519 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:17.488524 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 220.213 < min (86400 seconds) 2014-05-19 18:36:17.488565 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:17.488571 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:17.488595 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:17.810361 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:17.810435 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:17.857446 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:17.857502 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:18.370199 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:18.370278 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:18.488681 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:18.488702 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:18.488706 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:18.488708 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:18.488750 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:18.773712 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:18.773785 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:18.946048 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:19.083518 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:19.083586 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:19.340267 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.340291 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.340365 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.340741 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.340906 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.341033 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.341110 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.354954 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.354993 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.361983 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.367236 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.388767 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:19.392369 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.392390 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.410348 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.410371 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.418896 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.418895 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:19.418924 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.418947 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.418963 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:19.488794 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:19.488842 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.08 < max 0.5 = yes 2014-05-19 18:36:19.488855 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:19.488863 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:19.488858 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 222.213 < min (86400 seconds) 2014-05-19 18:36:19.488876 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:19.488881 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:19.488883 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:19.788874 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:19.788941 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:20.158037 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:20.158139 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:20.488964 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:20.488987 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:20.488993 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:20.488998 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:20.489024 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:20.670642 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:20.670707 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:20.773800 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:20.773863 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:20.946143 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:21.153587 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:21.153606 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:21.154030 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:21.154123 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:21.154159 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:21.154186 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:21.154235 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:21.154281 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:21.154357 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:21.154397 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:21.154419 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:21.154442 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:21.154468 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:21.154490 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:21.154514 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:21.154537 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:21.154555 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:21.154578 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:21.340339 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.340366 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.340457 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.340834 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.340975 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.341099 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.341182 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.355036 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.355072 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.362080 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.367332 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.388847 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:21.392468 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.392495 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.410448 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.410472 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.418996 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:21.418998 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.419065 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.419124 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.419149 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:21.489085 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:21.489107 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:21.489111 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:21.489113 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:21.489138 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:22.089337 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:22.089393 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:22.185766 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:22.185822 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:22.489199 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:22.489216 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:36:22.489220 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:36:22.489223 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:36:22.489231 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:22.489234 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:22.489236 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:22.489276 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:22.773876 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:22.773940 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:22.919779 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:22.919824 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:22.946237 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:22.971085 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:22.971160 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:23.069278 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:23.069457 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:23.110552 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:23.110651 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:23.340421 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.340449 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.340527 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.340915 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.341053 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.341171 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.341260 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.355119 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.355149 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.362165 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.367428 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.388936 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:23.392567 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.392621 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.410530 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.410550 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.419082 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:23.419098 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.419184 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.419224 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.419246 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:23.489324 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:23.489345 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:23.489350 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:23.489352 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:23.489375 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:24.486192 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:24.486306 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:24.489439 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:24.489487 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes 2014-05-19 18:36:24.489500 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:24.489508 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:24.489503 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 227.214 < min (86400 seconds) 2014-05-19 18:36:24.489522 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:24.489525 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:24.489527 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:24.654107 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:24.654125 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:24.654597 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:24.654662 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:24.654713 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:24.654740 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:24.654863 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:24.654923 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:24.654957 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:24.654980 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:24.655003 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:24.655028 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:24.655053 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:24.655074 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:24.655097 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:24.655119 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:24.655140 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:24.655165 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:24.773969 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:24.774016 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:24.946330 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:24.984033 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:24.984147 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:25.154647 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:25.154666 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:25.155174 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:25.155244 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:25.155340 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:25.155384 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:25.155409 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:25.155433 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:25.155452 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:25.155506 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:25.155563 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:25.155586 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:25.155611 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:25.155633 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:25.155661 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:25.155680 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:25.155710 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:25.155736 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:25.340495 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.340523 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.340592 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.340996 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.341118 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.341238 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.341341 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.355207 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.355227 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.362248 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.367526 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.389031 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:25.392679 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.392700 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.410628 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.410654 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.410936 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:25.411015 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:25.419169 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:25.419176 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.419272 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.419301 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.419313 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:25.489613 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:25.489665 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:25.489661 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes 2014-05-19 18:36:25.489675 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:25.489678 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 228.214 < min (86400 seconds) 2014-05-19 18:36:25.489697 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:25.489700 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:25.489702 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:26.058404 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:26.058456 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:26.420282 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:26.420369 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:26.489784 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:26.489832 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes 2014-05-19 18:36:26.489845 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:26.489853 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:26.489848 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 229.214 < min (86400 seconds) 2014-05-19 18:36:26.489866 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:26.489869 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:26.489871 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:26.569828 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:26.569943 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:26.774062 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:26.774095 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:26.920957 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:26.921045 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:26.946425 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:27.070437 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:27.070581 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:27.340595 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.340618 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.340654 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.341078 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.341190 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.341316 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.341413 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.355291 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.355309 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.362331 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.367622 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.389127 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:27.392778 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.392833 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.410727 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.410749 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.419256 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:27.419259 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.419353 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.419376 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.419402 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:27.489957 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:27.489974 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:36:27.489978 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:36:27.489982 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:36:27.490008 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:27.490016 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes 2014-05-19 18:36:27.490029 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:27.490032 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 230.214 < min (86400 seconds) 2014-05-19 18:36:27.490050 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:27.490053 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:27.490055 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:27.989754 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:27.989862 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:28.055205 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:28.055223 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:28.055667 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:28.055755 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:28.055815 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:28.055943 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:28.056006 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:28.056032 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:28.056063 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:28.056086 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:28.056108 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:28.056130 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:28.056157 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:28.056180 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:28.056206 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:28.056228 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:28.056250 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:28.056273 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:28.484545 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:28.484678 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:28.490141 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:28.490189 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes 2014-05-19 18:36:28.490202 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:28.490210 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:28.490205 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 231.214 < min (86400 seconds) 2014-05-19 18:36:28.490223 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:28.490227 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:28.490229 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:28.774161 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:28.774175 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:28.871306 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:28.871411 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:28.946519 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:28.985165 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:28.985232 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:29.340672 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.340696 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.340714 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.341150 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.341259 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.341377 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.341491 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.355384 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.355426 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.362408 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.367719 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.389224 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:29.392867 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.392944 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.410828 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.410888 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.419354 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:29.419355 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.419439 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.419465 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.419495 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:29.490315 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:29.490367 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:29.490363 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes 2014-05-19 18:36:29.490376 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:29.490379 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 232.214 < min (86400 seconds) 2014-05-19 18:36:29.490398 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:29.490401 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:29.490402 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:29.559090 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:29.559155 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:29.821638 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:29.821691 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:29.970986 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:29.971105 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:30.059524 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:30.059597 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:30.386448 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:30.386514 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:30.490486 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:30.490506 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:30.490511 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:30.490513 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:30.490542 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:30.774258 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:30.774260 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:30.946612 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:31.311288 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:31.311332 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:31.340755 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.340774 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.340802 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.341218 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.341339 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.341448 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.341587 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.355481 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.355502 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.362502 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.367823 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.389319 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:31.392948 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.393030 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.410926 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.410968 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.419461 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.419460 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:31.419519 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.419541 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.419566 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:31.490251 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:31.490307 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:31.490571 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:31.490627 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:31.490623 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes 2014-05-19 18:36:31.490651 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:31.490657 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 234.215 < min (86400 seconds) 2014-05-19 18:36:31.490682 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:31.490687 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:31.490691 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:31.885740 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:31.885836 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:31.990736 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:31.990820 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:32.371660 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:32.371751 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:32.490773 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:32.490791 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:36:32.490796 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:36:32.490802 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:36:32.490813 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:32.490819 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:32.490823 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:32.490851 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:32.755751 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:32.755769 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:32.756233 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:32.756313 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:32.756357 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:32.756415 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:32.756454 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:32.756551 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:32.756634 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:32.756677 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:32.756704 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:32.756724 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:32.756749 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:32.756784 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:32.756813 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:32.756834 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:32.756856 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:32.756879 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:32.774342 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:32.774347 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:32.872302 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:32.872359 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:32.946705 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:32.960014 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:32.960076 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:33.340833 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.340852 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.340896 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.341287 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.341406 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.341516 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.341666 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.355581 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.355603 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.362589 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.367919 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.389414 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:33.393029 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.393112 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.411010 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.411048 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.419567 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:33.419572 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.419598 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.419620 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.419644 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:33.490893 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:33.490914 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:33.490919 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:33.490921 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:33.490944 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:34.456326 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:34.456344 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:34.456849 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:34.456897 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:34.456923 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:34.456960 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:34.457024 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:34.457046 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:34.457116 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:34.457193 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:34.457214 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:34.457239 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:34.457262 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:34.457281 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:34.457301 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:34.457320 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:34.457341 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:34.457364 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:34.490992 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:34.491053 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:34.491049 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.06 < max 0.5 = yes 2014-05-19 18:36:34.491080 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:34.491086 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 237.215 < min (86400 seconds) 2014-05-19 18:36:34.491111 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:34.491117 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:34.491120 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:34.522119 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:34.522196 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:34.671568 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:34.671667 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:34.774434 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:34.774437 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:34.891296 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:34.891364 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:34.946798 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:35.340930 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.340959 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.340981 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.341356 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.341476 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.341582 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.341751 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.355681 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.355703 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.362682 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.368015 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.389508 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:35.393110 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.393198 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.411107 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.411127 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.419667 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:35.419672 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.419694 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.419717 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.419770 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:35.491190 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:35.491241 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:35.491238 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.06 < max 0.5 = yes 2014-05-19 18:36:35.491252 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:35.491255 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 238.215 < min (86400 seconds) 2014-05-19 18:36:35.491273 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:35.491276 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:35.491277 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:35.686737 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:35.686864 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:35.772713 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:35.772867 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:36.156872 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:36.156890 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:36.157378 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:36.157413 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:36.157455 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:36.157578 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:36.157621 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:36.157661 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:36.157684 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:36.157704 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:36.157723 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:36.157742 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:36.157767 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:36.157789 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:36.157813 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:36.157832 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:36.157851 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:36.157871 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:36.187232 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:36.187304 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:36.222625 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:36.222743 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:36.372009 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:36.372171 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:36.491358 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:36.491379 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:36.491384 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:36.491386 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:36.491416 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:36.586256 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:36.586310 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:36.611575 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:36.611633 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:36.687753 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:36.687847 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:36.774537 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:36.774540 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:36.946895 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:37.112013 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:37.112065 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:37.341029 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.341051 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.341074 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.341422 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.341546 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.341660 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.341840 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.355781 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.355803 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.362780 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.368110 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.389603 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:37.393191 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.393282 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.411209 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.411268 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.419766 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:37.419769 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.419807 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.419826 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.419852 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:37.491471 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:37.491488 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:36:37.491492 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:36:37.491496 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:36:37.491523 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:37.491530 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.06 < max 0.5 = yes 2014-05-19 18:36:37.491543 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:37.491546 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 240.216 < min (86400 seconds) 2014-05-19 18:36:37.491564 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:37.491567 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:37.491569 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:37.612499 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:37.612570 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:37.660621 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:37.660687 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:37.923230 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:37.923314 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:38.072548 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:38.072673 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:38.286822 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:38.286885 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:38.491651 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:38.491672 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:38.491676 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:38.491679 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:38.491708 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:38.774644 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:38.774648 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:38.946962 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:39.341104 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.341130 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.341147 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.341487 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.341625 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.341727 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.341920 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.355881 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.355903 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.361198 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:39.361308 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:39.362867 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.368207 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.389698 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:39.393277 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.393366 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.411307 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.411348 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.419874 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:39.419877 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.419912 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.419931 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.419955 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:39.491763 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:39.491815 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:39.491812 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.06 < max 0.5 = yes 2014-05-19 18:36:39.491825 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:39.491828 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 242.216 < min (86400 seconds) 2014-05-19 18:36:39.491847 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:39.491850 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:39.491852 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:39.591766 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:39.591821 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:39.987386 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:39.987556 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:40.257425 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:40.257443 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:40.257913 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:40.257964 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:40.258062 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:40.258191 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:40.258221 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:40.258252 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:40.258286 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:40.258306 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:40.258325 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:40.258343 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:40.258367 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:40.258390 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:40.258412 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:40.258432 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:40.258451 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:40.258470 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:40.472968 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:40.473068 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:40.491923 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:40.491944 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:40.491949 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:40.491951 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:40.491975 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:40.774738 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:40.774741 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:40.788093 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:40.788161 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:40.947039 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:41.061668 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:41.061784 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:41.292259 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:41.292369 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:41.341203 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.341235 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.341256 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.341552 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.341698 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.341800 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.341990 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.355964 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.355986 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.358017 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:41.358035 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:41.358465 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:41.358519 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:41.358552 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:41.358572 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:41.358718 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:41.358772 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:41.358820 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:41.358878 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:41.358901 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:41.358923 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:41.358951 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:41.358973 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:41.359001 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:41.359022 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:41.359044 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:41.359067 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:41.362948 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.368287 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.389793 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:41.393372 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.393449 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.411404 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.411426 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.419971 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:41.419973 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.420012 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.420031 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.420060 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:41.492023 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:41.492043 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:41.492048 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:41.492050 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:41.492092 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:41.712891 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:41.712983 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:42.023730 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:42.023784 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:42.173196 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:42.173294 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:42.173381 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:42.173589 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:42.492118 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:42.492136 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:36:42.492141 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:36:42.492147 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:36:42.492159 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:42.492165 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:42.492168 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:42.492197 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:42.774827 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:42.774830 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:42.947107 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:42.992845 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:42.992915 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:43.058569 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:43.058587 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:43.059029 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:43.059073 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:43.059099 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:43.059158 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:43.059188 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:43.059248 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:43.059306 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:43.059347 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:43.059389 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:43.059420 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:43.059448 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:43.059470 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:43.059495 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:43.059516 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:43.059538 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:43.059561 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:43.124127 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:43.124262 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:43.273767 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:43.273879 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:43.341303 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.341324 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.341348 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.341617 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.341767 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.341871 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.342068 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.356050 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.356068 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.363028 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.368386 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.389891 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:43.393470 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.393534 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.411507 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.411564 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.420064 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:43.420068 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.420112 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.420134 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.420160 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:43.492234 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:43.492255 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:43.492259 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:43.492261 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:43.492285 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:43.873763 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:43.873845 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:44.087992 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:44.088124 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:44.492325 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:44.492373 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.05 < max 0.5 = yes 2014-05-19 18:36:44.492386 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:44.492382 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:44.492389 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 247.216 < min (86400 seconds) 2014-05-19 18:36:44.492409 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:44.492412 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:44.492414 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:44.774914 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:44.774919 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:44.824782 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:44.824830 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:44.888487 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:44.888548 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:44.947182 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:44.974228 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:44.974276 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:45.162276 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:45.162341 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:45.188489 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:45.188605 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:45.341395 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.341416 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.341434 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.341681 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.341836 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.341940 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.342137 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.356137 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.356154 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.363109 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.368481 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.389986 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:45.393567 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.393612 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.411603 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.411642 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.420159 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:45.420162 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.420190 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.420207 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.420225 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:45.492485 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:45.492506 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:45.492511 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:45.492513 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:45.492537 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:45.813243 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:45.813310 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:46.262800 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:46.262857 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:46.492584 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:46.492604 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:46.492609 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:46.492611 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:46.492641 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:46.775011 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:46.775013 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:46.889091 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:46.889155 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:46.947267 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:47.093291 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:47.093368 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:47.341494 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.341525 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.341547 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.341745 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.341903 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.342010 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.342206 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.356229 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.356251 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.363190 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.368593 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.390082 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:47.393664 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.393692 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.411700 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.411719 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.420276 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.420276 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:47.420305 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.420363 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.420387 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:47.492679 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:47.492696 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:36:47.492700 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:36:47.492704 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:36:47.492711 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:47.492715 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:47.492717 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:47.492755 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:47.963242 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:47.963332 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:47.974083 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:47.974208 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:48.193871 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:48.193929 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:48.492796 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:48.492816 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:48.492821 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:48.492823 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:48.492853 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:48.775097 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:48.775106 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:48.947353 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:48.959106 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:48.959125 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:48.959567 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:48.959731 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:48.959791 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:48.959836 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:48.959877 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:48.959922 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:48.959965 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:48.960028 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:48.960073 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:48.960116 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:48.960162 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:48.960204 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:48.960251 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:48.960293 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:48.960333 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:48.960377 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:49.074521 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:49.074585 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:49.341594 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.341615 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.341634 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.341810 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.341973 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.342080 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.342276 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.356312 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.356337 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.363279 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.368689 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.390179 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:49.393772 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.393830 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.411801 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.411859 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.420376 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:49.420379 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.420406 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.420434 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.420467 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:49.492910 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:49.492931 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:49.492936 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:49.492938 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:49.492968 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:49.894404 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:49.894511 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:50.188905 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:50.188975 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:50.493022 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:50.493071 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.05 < max 0.5 = yes 2014-05-19 18:36:50.493083 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:50.493079 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:50.493087 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 253.217 < min (86400 seconds) 2014-05-19 18:36:50.493105 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:50.493108 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:50.493110 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:50.725202 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:50.725281 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:50.774935 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:50.775018 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:50.775161 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:50.775172 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:50.874863 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:50.874958 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:50.947433 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:51.113575 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:51.113697 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:51.341693 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.341713 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.341731 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.341874 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.342042 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.342150 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.342348 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.356399 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.356419 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.363375 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.368769 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.390275 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:51.393856 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.393912 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.411884 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.411941 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.420463 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:51.420470 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.420488 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.420513 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.420544 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:51.493187 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:51.493208 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:51.493213 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:51.493215 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:51.493245 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:51.889286 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:51.889369 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:52.493276 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:52.493293 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:36:52.493297 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:36:52.493301 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:36:52.493309 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:52.493312 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:52.493314 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:52.493345 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:52.775238 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:52.775240 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:52.789699 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:52.789781 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:52.813921 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:52.814106 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:52.947517 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:52.989761 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:52.989903 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:53.341785 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.341837 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.341860 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.341937 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.342101 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.342220 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.342416 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.356487 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.356508 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.363464 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.368866 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.390371 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:53.393939 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.393989 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.411976 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.412019 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.420549 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:53.420554 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.420572 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.420615 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.420639 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:53.493398 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:53.493446 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.04 < max 0.5 = yes 2014-05-19 18:36:53.493459 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:53.493456 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:53.493462 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 256.217 < min (86400 seconds) 2014-05-19 18:36:53.493481 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:53.493484 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:53.493486 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:53.659660 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:53.659678 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:53.660096 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:53.660142 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:53.660236 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:53.660307 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:53.660443 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:53.660488 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:53.660539 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:53.660606 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:53.660654 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:53.660693 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:53.660744 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:53.660788 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:53.660833 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:53.660874 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:53.660916 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:53.660958 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:53.863684 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:53.863778 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:53.914395 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:53.914510 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:54.493568 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:54.493620 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:54.493617 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.04 < max 0.5 = yes 2014-05-19 18:36:54.493630 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:54.493633 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 257.218 < min (86400 seconds) 2014-05-19 18:36:54.493651 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:54.493654 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:54.493656 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:54.690191 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:54.690291 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:54.775329 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:54.775332 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:54.947614 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:55.341881 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.341905 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.341933 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.342000 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.342165 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.342292 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.342483 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.356594 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.356620 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.363556 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.368961 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.390470 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:55.394021 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.394068 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.412059 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.412099 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.420631 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:55.420634 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.420664 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.420680 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.420717 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:55.425642 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:55.425699 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:55.493740 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:55.493761 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:55.493765 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:55.493768 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:55.493790 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:55.575422 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:55.575497 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:55.614778 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:55.614876 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:55.794738 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:55.794801 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:56.493819 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:56.493839 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:56.493844 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:56.493846 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:56.493887 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:56.675233 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:56.675290 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:56.775417 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:56.775418 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:56.947710 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:57.341973 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.342027 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.342050 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.342077 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.342231 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.342358 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.342549 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.356688 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.356723 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.363620 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.369058 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.390566 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:57.394107 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.394148 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.412143 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.412180 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.420727 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:57.420732 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.420752 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.420784 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.420809 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:57.490269 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:57.490357 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:57.493917 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:57.493935 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:36:57.493939 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:36:57.493942 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:36:57.493951 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:57.493954 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:57.493956 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:57.493968 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:58.494040 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:58.494061 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:36:58.494065 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:58.494068 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:36:58.494091 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:58.564202 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:58.564281 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:58.775510 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:58.775515 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:36:58.947782 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:36:58.960185 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:58.960204 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:36:58.960786 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:58.960823 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:36:58.960847 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:58.960867 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:36:58.960887 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:58.960909 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:58.960945 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:36:58.960970 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:36:58.961019 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:58.961066 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:36:58.961107 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:58.961132 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:58.961157 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:36:58.961179 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:58.961200 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:36:58.961222 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:36:59.342069 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.342125 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.342145 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.342159 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.342299 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.342418 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.342615 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.356790 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.356824 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.363711 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.369154 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.390661 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:36:59.394206 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.394228 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.412227 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.412258 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.420825 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:36:59.420829 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.420863 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.420889 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.420939 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:36:59.494152 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:36:59.494200 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.04 < max 0.5 = yes 2014-05-19 18:36:59.494213 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:36:59.494220 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:36:59.494216 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 262.218 < min (86400 seconds) 2014-05-19 18:36:59.494234 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:36:59.494237 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:36:59.494239 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:00.060751 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:00.060769 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:00.061262 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:00.061299 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:00.061347 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:00.061483 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:00.061526 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:00.061589 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:00.061628 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:00.061654 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:00.061677 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:00.061698 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:00.061723 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:00.061745 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:00.061771 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:00.061793 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:00.061814 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:00.061837 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:00.494310 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:00.494332 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:00.494336 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:00.494339 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:00.494362 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:00.495190 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:00.495256 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:00.590518 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:00.590618 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:00.726125 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:00.726183 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:00.775602 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:00.775604 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:00.875926 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:00.875989 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:00.947863 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:01.342169 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.342203 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.342227 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.342252 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.342369 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.342484 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.342692 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.356894 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.356925 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.363794 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.369235 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.375561 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:01.375623 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:01.390757 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:01.394307 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.394368 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.412308 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.412338 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.420918 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:01.420924 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.420943 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.420967 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.421015 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:01.494423 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:01.494471 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.04 < max 0.5 = yes 2014-05-19 18:37:01.494485 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:01.494492 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:01.494488 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 264.219 < min (86400 seconds) 2014-05-19 18:37:01.494507 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:01.494510 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:01.494512 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:01.515079 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:01.515144 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:01.826608 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:01.826710 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:01.976500 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:01.976714 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:02.494595 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:02.494612 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:37:02.494616 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:37:02.494619 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:37:02.494627 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:02.494630 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:02.494633 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:02.494671 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:02.775673 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:02.775695 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:02.790796 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:02.790955 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:02.947932 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:02.961252 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:02.961270 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:02.961744 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:02.961789 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:02.961815 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:02.961857 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:02.961906 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:02.961948 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:02.962004 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:02.962054 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:02.962075 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:02.962094 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:02.962117 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:02.962142 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:02.962169 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:02.962191 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:02.962213 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:02.962236 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:03.342263 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.342284 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.342295 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.342323 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.342433 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.342549 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.342765 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.356987 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.357020 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.363875 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.369331 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.390854 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:03.394406 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.394455 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.412394 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.412416 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.421018 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:03.421023 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.421106 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.421156 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.421215 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:03.461848 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:03.461867 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:03.462332 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:03.462382 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:03.462426 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:03.462461 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:03.462505 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:03.462554 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:03.462625 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:03.462668 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:03.462717 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:03.462763 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:03.462813 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:03.462859 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:03.462906 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:03.462949 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:03.462991 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:03.463030 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:03.494703 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:03.494752 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.04 < max 0.5 = yes 2014-05-19 18:37:03.494763 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:03.494768 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:03.494771 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 266.219 < min (86400 seconds) 2014-05-19 18:37:03.494790 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:03.494793 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:03.494794 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:03.864880 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:03.864963 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:03.891346 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:03.891413 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:04.494873 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:04.494895 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:04.494899 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:04.494902 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:04.494942 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:04.727051 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:04.727160 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:04.775751 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:04.775774 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:04.877029 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:04.877188 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:04.948015 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:04.965424 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:04.965595 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:05.227675 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:05.227769 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:05.290724 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:05.290810 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:05.342362 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.342383 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.342410 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.342426 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.342497 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.342612 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.342833 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.357089 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.357114 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.363959 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.369428 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.377585 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:05.377681 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:05.390935 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:05.394488 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.394532 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.412477 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.412499 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.421133 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:05.421154 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.421178 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.421241 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.421283 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:05.494988 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:05.495009 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:05.495014 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:05.495016 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:05.495039 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:05.795602 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:05.795660 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:06.215475 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:06.215556 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:06.495103 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:06.495124 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:06.495129 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:06.495131 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:06.495172 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:06.675861 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:06.675955 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:06.775830 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:06.775852 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:06.791956 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:06.792088 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:06.896222 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:06.896273 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:06.948095 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:06.962411 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:06.962430 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:06.962938 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:06.963011 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:06.963083 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:06.963138 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:06.963207 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:06.963260 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:06.963283 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:06.963371 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:06.963414 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:06.963439 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:06.963467 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:06.963500 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:06.963530 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:06.963552 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:06.963575 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:06.963598 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:07.292412 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:07.292485 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:07.342459 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.342479 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.342498 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.342525 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.342562 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.342677 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.342901 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.357190 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.357214 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.364042 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.369526 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.391031 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:07.394583 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.394613 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.412563 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.412606 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.421233 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:07.421235 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.421269 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.421320 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.421353 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:07.495215 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:07.495232 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:37:07.495236 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:37:07.495239 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:37:07.495247 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:07.495251 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:07.495253 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:07.495282 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:07.776319 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:07.776394 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:07.865910 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:07.865964 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:08.191134 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:08.191235 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:08.366481 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:08.366547 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:08.495338 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:08.495358 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:08.495363 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:08.495365 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:08.495406 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:08.662948 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:08.662966 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:08.663424 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:08.663473 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:08.663508 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:08.663554 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:08.663703 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:08.663750 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:08.663793 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:08.663818 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:08.663841 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:08.663865 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:08.663891 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:08.663916 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:08.663941 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:08.663966 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:08.663991 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:08.664018 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:08.728249 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:08.728333 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:08.775908 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:08.775931 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:08.878184 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:08.878333 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:08.948174 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:09.115807 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:09.115866 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:09.342558 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.342579 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.342594 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.342617 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.342628 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.342740 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.342966 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.357292 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.357334 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.364126 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.369623 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.391127 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:09.394665 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.394693 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.412681 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.412707 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.421345 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:09.421348 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.421403 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.421419 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.421446 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:09.495450 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:09.495471 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:09.495476 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:09.495478 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:09.495519 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:09.796872 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:09.796941 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:09.891566 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:09.891621 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:10.297382 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:10.297469 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:10.428772 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:10.428844 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:10.495559 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:10.495609 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.03 < max 0.5 = yes 2014-05-19 18:37:10.495622 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:10.495628 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:10.495626 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 273.22 < min (86400 seconds) 2014-05-19 18:37:10.495644 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:10.495647 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:10.495649 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:10.578721 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:10.578798 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:10.676684 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:10.676731 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:10.775997 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:10.776004 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:10.792938 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:10.793076 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:10.816254 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:10.816373 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:10.948255 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:10.991995 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:10.992077 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:11.177163 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:11.177232 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:11.342655 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.342678 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.342729 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.342752 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.342804 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.342820 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.343030 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.357389 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.357408 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.364210 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.369704 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.391223 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:11.394764 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.394790 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.412782 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.412809 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.421453 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:11.421457 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.421497 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.421523 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.421547 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:11.495719 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:11.495740 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:11.495745 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:11.495747 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:11.495787 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:11.866948 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:11.867028 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:11.916725 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:11.916769 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:12.493443 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:12.493522 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:12.495784 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:12.495797 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:37:12.495800 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:37:12.495803 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:37:12.495809 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:12.495812 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:12.495814 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:12.495849 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:12.776088 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:12.776092 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:12.948339 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:13.342746 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.342771 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.342801 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.342822 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.342867 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.342896 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.343105 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.357496 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.357522 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.364293 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.369802 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.391318 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:13.394851 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.394872 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.412882 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.412906 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.421563 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:13.421564 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.421636 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.421659 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.421683 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:13.495882 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:13.495931 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.03 < max 0.5 = yes 2014-05-19 18:37:13.495944 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:13.495951 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:13.495948 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 276.22 < min (86400 seconds) 2014-05-19 18:37:13.495967 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:13.495971 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:13.495972 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:13.567470 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:13.567541 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:13.797853 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:13.797941 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:13.892255 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:13.892311 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:13.963469 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:13.963488 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:13.963962 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:13.964008 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:13.964082 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:13.964167 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:13.964219 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:13.964267 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:13.964303 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:13.964339 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:13.964380 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:13.964421 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:13.964467 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:13.964511 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:13.964556 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:13.964630 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:13.964675 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:13.964718 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:14.392732 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:14.392795 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:14.496046 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:14.496068 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:14.496072 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:14.496074 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:14.496115 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:14.677541 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:14.677598 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:14.776180 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:14.776182 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:14.817072 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:14.817116 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:14.948419 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:15.317494 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:15.317618 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:15.342847 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.342869 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.342891 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.342911 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.342937 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.342959 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.343191 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.357589 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.357613 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.364374 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.369900 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.391414 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:15.394938 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.394957 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.412983 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.413009 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.421669 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:15.421672 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.421725 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.421740 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.421755 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:15.496158 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:15.496206 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.03 < max 0.5 = yes 2014-05-19 18:37:15.496219 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:15.496215 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:15.496222 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 278.22 < min (86400 seconds) 2014-05-19 18:37:15.496241 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:15.496243 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:15.496245 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:15.498355 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:15.498418 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:15.729216 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:15.729265 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:15.879274 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:15.879386 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:16.377955 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:16.378039 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:16.496325 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:16.496347 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:16.496351 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:16.496354 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:16.496395 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:16.776281 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:16.776282 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:16.948503 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:17.342938 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.342960 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.342980 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.343001 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.343038 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.343059 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.343254 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.357644 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.357680 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.364459 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.369997 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.391509 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:17.395039 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.395065 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.413082 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.413106 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.421775 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:17.421781 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.421814 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.421840 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.421856 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:17.496438 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:17.496454 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:37:17.496458 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:37:17.496462 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:37:17.496470 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:17.496474 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:17.496475 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:17.496514 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:17.794029 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:17.794088 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:17.893037 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:17.893096 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:18.496559 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:18.496594 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:18.496599 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:18.496601 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:18.496628 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:18.776358 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:18.776375 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:18.817820 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:18.817884 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:18.867849 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:18.867900 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:18.948590 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:19.343035 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.343058 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.343085 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.343120 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.343145 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.343193 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.343310 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.357742 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.357773 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.364550 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.370095 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.391589 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:19.395139 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.395168 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.413180 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.413207 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.421873 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:19.421877 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.421915 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.421938 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.421962 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:19.496685 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:19.496706 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:19.496711 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:19.496713 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:19.496737 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:19.593510 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:19.593568 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:19.864017 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:19.864036 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:19.864493 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:19.864568 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:19.864645 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:19.864768 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:19.864801 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:19.864821 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:19.864856 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:19.864877 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:19.864897 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:19.864919 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:19.864943 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:19.864966 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:19.864989 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:19.865011 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:19.865033 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:19.865058 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:20.496781 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:20.496830 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.03 < max 0.5 = yes 2014-05-19 18:37:20.496843 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:20.496851 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:20.496846 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 283.221 < min (86400 seconds) 2014-05-19 18:37:20.496865 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:20.496868 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:20.496870 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:20.518263 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:20.518353 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:20.776455 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:20.776457 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:20.798804 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:20.798844 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:20.948680 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:21.343132 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.343156 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.343173 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.343194 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.343208 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.343263 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.343374 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.357818 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.357846 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.364646 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.370191 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.391669 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:21.395239 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.395263 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.413280 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.413304 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.421980 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:21.421983 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.422030 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.422053 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.422075 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:21.496955 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:21.496976 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:21.496980 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:21.496982 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:21.497012 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:21.629622 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:21.629786 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:21.678265 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:21.678317 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:21.779827 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:21.779888 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:22.497066 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:22.497084 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:37:22.497089 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:37:22.497095 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:37:22.497145 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:22.497138 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.03 < max 0.5 = yes 2014-05-19 18:37:22.497156 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:22.497161 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 285.221 < min (86400 seconds) 2014-05-19 18:37:22.497197 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:22.497203 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:22.497206 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:22.776542 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:22.776550 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:22.948760 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:23.343233 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.343257 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.343279 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.343294 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.343313 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.343358 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.343429 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.357902 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.357928 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.364743 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.370288 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.391751 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:23.395340 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.395367 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.413381 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.413406 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.422087 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:23.422094 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.422114 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.422134 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.422144 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:23.497294 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:23.497314 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:23.497319 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:23.497321 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:23.497351 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:23.694538 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:23.694619 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:24.497398 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:24.497418 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:24.497423 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:24.497425 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:24.497466 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:24.768313 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:24.768374 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:24.776640 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:24.776643 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:24.893791 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:24.893862 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:24.948851 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:25.343328 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.343351 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.343366 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.343388 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.343409 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.343429 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.343492 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.357978 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.358006 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.364840 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.370385 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.391831 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:25.395438 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.395462 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.413464 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.413485 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.422182 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.422181 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:25.422214 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.422233 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.422254 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:25.497509 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:25.497530 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:25.497535 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:25.497537 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:25.497578 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:25.764550 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:25.764568 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:25.765126 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:25.765197 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:25.765247 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:25.765299 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:25.765383 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:25.765435 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:25.765466 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:25.765508 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:25.765531 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:25.765550 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:25.765572 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:25.765591 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:25.765613 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:25.765638 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:25.765660 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:25.765685 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:25.818553 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:25.818609 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:26.497619 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:26.497642 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:26.497649 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:26.497653 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:26.497680 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:26.699240 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:26.699340 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:26.776724 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:26.776727 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:26.948946 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:27.343426 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.343453 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.343473 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.343491 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.343509 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.343537 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.343556 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.358060 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.358083 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.364938 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.370482 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.391912 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:27.395523 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.395542 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.413547 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.413567 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.422264 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.422272 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:27.422281 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.422298 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.422334 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:27.497726 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:27.497743 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:37:27.497747 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:37:27.497751 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:37:27.497753 7fd04f907700 10 osd.0 21 send_pg_stats - 0 pgs updated 2014-05-19 18:37:27.497786 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:27.497792 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:27.497795 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:27.497823 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:27.498210 7fd04aecb700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:27.498214 7fd04aecb700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:27.498216 7fd04aecb700 20 osd.0 21 _dispatch 0x7890380 pg_stats_ack(0 pgs tid 8) v1 2014-05-19 18:37:27.498219 7fd04aecb700 10 osd.0 21 handle_pg_stats_ack 2014-05-19 18:37:27.498221 7fd04aecb700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:27.498223 7fd04aecb700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:27.530034 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:27.530088 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:27.578618 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:27.578689 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:27.680383 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:27.680603 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:28.497881 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:28.497903 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:28.497907 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:28.497910 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:28.497950 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:28.776824 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:28.776828 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:28.949038 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:29.343479 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.343525 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.343543 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.343557 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.343582 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.343619 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.343637 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.358116 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.358160 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.365037 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.370579 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.392002 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:29.395608 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.395627 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.413630 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.413650 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.422360 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:29.422365 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.422410 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.422432 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.422456 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:29.497994 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:29.498015 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:29.498019 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:29.498022 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:29.498063 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:29.595129 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:29.595194 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:30.498105 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:30.498125 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:30.498130 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:30.498132 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:30.498162 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:30.668778 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:30.668861 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:30.776924 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:30.776926 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:30.794060 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:30.794125 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:30.949132 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:31.343552 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.343598 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.343622 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.343634 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.343663 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.343706 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.343718 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.358193 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.358240 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.365135 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.370660 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.392098 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:31.395706 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.395733 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.413730 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.413756 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.422470 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.422470 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:31.422508 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.422537 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.422569 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:31.498218 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:31.498239 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:31.498243 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:31.498245 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:31.498287 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:31.665177 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:31.665195 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:31.665779 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:31.665822 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:31.665849 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:31.665882 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:31.665970 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:31.666015 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:31.666053 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:31.666097 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:31.666150 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:31.666194 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:31.666219 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:31.666238 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:31.666264 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:31.666286 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:31.666307 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:31.666330 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:31.718790 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:31.718879 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:32.498303 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:32.498320 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:37:32.498323 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:37:32.498327 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:37:32.498335 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:32.498338 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:32.498340 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:32.498371 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:32.599585 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:32.599641 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:32.777019 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:32.777020 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:32.949221 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:33.343633 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.343659 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.343684 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.343708 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.343721 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.343765 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.343783 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.358253 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.358309 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.365232 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.370757 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.392192 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:33.395792 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.395812 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.413812 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.413834 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.422574 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:33.422577 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.422613 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.422634 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.422649 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:33.430548 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:33.430654 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:33.479094 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:33.479152 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:33.498412 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:33.498433 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:33.498438 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:33.498440 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:33.498469 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:33.581014 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:33.581075 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:33.965873 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:33.965891 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:33.966417 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:33.966453 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:33.966485 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:33.966505 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:33.966580 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:33.966634 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:33.966712 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:33.966786 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:33.966813 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:33.966832 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:33.966855 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:33.966877 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:33.966901 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:33.966923 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:33.966945 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:33.966967 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:34.498508 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:34.498567 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:34.498565 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes 2014-05-19 18:37:34.498596 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:34.498601 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 297.223 < min (86400 seconds) 2014-05-19 18:37:34.498627 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:34.498632 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:34.498636 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:34.777118 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:34.777122 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:34.949301 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:35.343721 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.343749 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.343772 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.343796 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.343818 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.343831 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.343885 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.358331 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.358382 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.365328 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.370855 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.392288 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:35.395875 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.395895 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.413908 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.413931 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.422682 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:35.422688 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.422720 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.422743 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.422760 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:35.495616 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:35.495724 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:35.498702 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:35.498718 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:35.498721 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:35.498723 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:35.498757 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:35.730972 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:35.731035 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:35.881500 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:35.881593 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:36.498786 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:36.498807 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:36.498812 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:36.498814 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:36.498843 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:36.569278 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:36.569354 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:36.694342 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:36.694424 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:36.777226 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:36.777229 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:36.949377 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:37.343802 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.343825 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.343847 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.343864 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.343874 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.343884 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.343990 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.358398 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.358480 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.365425 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.370954 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.392384 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:37.395959 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.395978 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.414008 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.414034 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.422794 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:37.422797 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.422908 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.422966 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.422988 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:37.498889 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:37.498906 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:37:37.498910 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:37:37.498914 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:37:37.498922 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:37.498926 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:37.498928 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:37.498968 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:37.619112 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:37.619214 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:37.796299 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:37.796408 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:38.066462 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:38.066481 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:38.066919 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:38.066964 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:38.066990 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:38.067085 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:38.067142 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:38.067222 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:38.067260 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:38.067295 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:38.067315 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:38.067338 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:38.067360 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:38.067378 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:38.067415 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:38.067435 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:38.067453 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:38.067472 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:38.499010 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:38.499032 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:38.499036 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:38.499039 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:38.499080 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:38.500123 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:38.500234 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:38.777313 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:38.777321 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:38.869779 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:38.869871 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:38.949470 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:39.343881 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.343903 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.343925 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.343943 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.343956 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.343965 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.344058 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.358478 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.358571 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.365521 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.371051 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.379459 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:39.379531 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:39.392481 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:39.396042 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.396063 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.414089 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.414111 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.422907 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:39.422931 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.422980 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.423057 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.423080 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:39.499125 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:39.499146 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:39.499151 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:39.499153 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:39.499194 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:39.831453 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:39.831566 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:39.982096 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:39.982229 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:40.499239 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:40.499299 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:40.499296 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes 2014-05-19 18:37:40.499334 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:40.499340 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 303.223 < min (86400 seconds) 2014-05-19 18:37:40.499367 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:40.499372 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:40.499376 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:40.777382 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:40.777400 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:40.800630 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:40.800707 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:40.949559 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:41.343961 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.343984 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.344004 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.344020 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.344045 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.344058 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.344119 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.358551 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.358651 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.365618 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.371147 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.392597 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:41.396126 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.396146 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.414171 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.414192 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.423009 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:41.423011 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.423042 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.423132 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.423145 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:41.499458 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:41.499506 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes 2014-05-19 18:37:41.499519 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:41.499528 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:41.499522 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 304.224 < min (86400 seconds) 2014-05-19 18:37:41.499541 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:41.499544 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:41.499546 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:41.679955 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:41.680050 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:41.896850 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:41.896915 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:42.167005 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:42.167023 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:42.167466 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:42.167509 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:42.167543 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:42.167597 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:42.167643 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:42.167694 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:42.167721 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:42.167776 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:42.167918 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:42.167947 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:42.167984 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:42.168005 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:42.168028 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:42.168047 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:42.168070 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:42.168095 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:42.499620 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:42.499637 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:37:42.499641 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:37:42.499645 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:37:42.499653 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:42.499657 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:42.499659 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:42.499698 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:42.594661 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:42.594720 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:42.777467 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:42.777478 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:42.949647 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:42.970225 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:42.970331 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:43.344059 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.344084 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.344104 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.344128 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.344144 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.344180 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.344193 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.358641 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.358726 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.365715 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.371244 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.392692 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:43.396206 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.396227 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.414255 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.414275 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.423115 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:43.423119 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.423183 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.423217 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.423241 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:43.499738 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:43.499759 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:43.499763 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:43.499766 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:43.499807 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:43.519444 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:43.519520 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:43.931842 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:43.931946 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:44.082685 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:44.082744 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:44.499827 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:44.499875 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes 2014-05-19 18:37:44.499888 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:44.499884 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:44.499891 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 307.224 < min (86400 seconds) 2014-05-19 18:37:44.499910 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:44.499913 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:44.499915 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:44.777548 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:44.777561 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:44.895016 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:44.895104 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:44.901050 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:44.901139 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:44.949745 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:45.344146 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.344168 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.344192 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.344204 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.344224 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.344243 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.344293 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.358723 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.358800 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.365815 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.371342 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.392787 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:45.396290 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.396310 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.414336 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.414358 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.423222 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:45.423226 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.423270 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.423295 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.423316 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:45.499984 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:45.500032 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes 2014-05-19 18:37:45.500045 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:45.500053 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:45.500048 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 308.224 < min (86400 seconds) 2014-05-19 18:37:45.500067 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:45.500070 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:45.500072 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:45.780418 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:45.780473 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:45.819866 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:45.819926 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:45.997335 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:45.997459 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:46.500139 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:46.500187 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes 2014-05-19 18:37:46.500200 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:46.500208 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:46.500203 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 309.224 < min (86400 seconds) 2014-05-19 18:37:46.500222 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:46.500225 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:46.500227 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:46.777648 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:46.777650 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:46.949838 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:47.070710 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:47.070768 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:47.344242 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.344260 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.344274 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.344292 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.344307 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.344326 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.344364 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.358818 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.358870 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.365912 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.371429 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.392883 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:47.396391 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.396417 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.414417 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.414435 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.423312 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:47.423318 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.423349 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.423371 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.423393 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:47.467559 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:47.467577 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:47.468048 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:47.468116 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:47.468212 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:47.468244 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:47.468271 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:47.468311 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:47.468365 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:47.468410 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:47.468449 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:47.468474 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:47.468501 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:47.468524 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:47.468550 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:47.468573 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:47.468618 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:47.468639 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:47.500296 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:47.500313 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:37:47.500317 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:37:47.500321 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:37:47.500336 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:47.500355 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes 2014-05-19 18:37:47.500367 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:47.500370 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 310.224 < min (86400 seconds) 2014-05-19 18:37:47.500389 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:47.500392 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:47.500394 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:48.500433 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:48.500454 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:48.500459 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:48.500461 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:48.500485 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:48.777755 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:48.777754 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:48.949931 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:48.995343 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:48.995455 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:49.001510 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:49.001569 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:49.232255 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:49.232366 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:49.344333 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.344361 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.344380 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.344399 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.344429 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.344446 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.344471 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.358909 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.358945 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.366009 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.371510 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.383274 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:49.383362 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:49.392982 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:49.396490 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.396514 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.414498 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.414521 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.423398 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:49.423403 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.423445 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.423456 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.423467 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:49.500543 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:49.500565 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:49.500569 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:49.500572 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:49.500613 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:49.880864 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:49.880982 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:49.920270 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:49.920368 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:50.500676 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:50.500724 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes 2014-05-19 18:37:50.500737 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:50.500733 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:50.500740 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 313.225 < min (86400 seconds) 2014-05-19 18:37:50.500759 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:50.500761 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:50.500763 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:50.777837 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:50.777852 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:50.950025 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:51.297959 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:51.298074 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:51.344429 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.344451 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.344474 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.344492 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.344522 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.344547 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.344566 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.359012 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.359040 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.366107 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.371591 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.393079 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:51.396590 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.396614 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.414597 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.414619 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.423496 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:51.423499 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.423542 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.423563 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.423586 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:51.500833 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:51.500854 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:51.500858 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:51.500860 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:51.500889 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:51.568137 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:51.568155 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:51.568718 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:51.568770 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:51.568799 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:51.568824 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:51.568847 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:51.568875 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:51.568915 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:51.568952 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:51.568977 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:51.569019 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:51.569044 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:51.569065 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:51.569087 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:51.569107 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:51.569131 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:51.569153 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:52.371277 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:52.371385 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:52.500940 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:52.500957 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:37:52.500961 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:37:52.500964 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:37:52.500972 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:52.500975 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:52.500978 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:52.501009 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:52.777933 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:52.777937 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:52.950107 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:53.095673 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:53.095796 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:53.332706 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:53.332755 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:53.344529 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.344552 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.344573 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.344603 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.344620 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.344643 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.344697 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.359112 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.359138 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.366203 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.371672 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.393175 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:53.396690 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.396719 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.414696 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.414720 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.423590 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:53.423594 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.423622 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.423639 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.423694 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:53.483997 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:53.484057 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:53.501048 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:53.501096 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes 2014-05-19 18:37:53.501109 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:53.501117 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:53.501112 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 316.225 < min (86400 seconds) 2014-05-19 18:37:53.501131 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:53.501134 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:53.501135 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:53.868779 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:53.868797 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:53.869332 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:53.869386 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:53.869434 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:53.869466 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:53.869497 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:53.869555 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:53.869617 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:53.869646 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:53.869683 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:53.869709 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:53.869735 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:53.869754 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:53.869774 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:53.869793 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:53.869814 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:53.869837 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:54.020554 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:54.020787 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:54.301900 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:54.302030 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:54.501219 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:54.501242 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:54.501248 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:54.501253 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:54.501280 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:54.778022 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:54.778022 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:54.950194 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:55.181152 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:55.181225 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:55.344624 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.344650 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.344672 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.344696 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.344720 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.344742 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.344767 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.359210 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.359233 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.366301 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.371752 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.393271 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:55.396791 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.396817 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.398519 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:55.398587 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:55.414794 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.414817 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.423686 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:55.423690 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.423713 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.423734 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.423760 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:55.501339 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:55.501387 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes 2014-05-19 18:37:55.501401 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:55.501408 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:55.501404 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 318.225 < min (86400 seconds) 2014-05-19 18:37:55.501422 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:55.501425 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:55.501427 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:55.633137 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:55.633192 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:55.784417 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:55.784543 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:56.471842 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:56.471958 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:56.501513 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:56.501534 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:56.501539 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:56.501541 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:56.501570 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:56.769400 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:56.769418 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:37:56.769863 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:56.769955 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:56.770023 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:56.770069 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:56.770114 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:56.770154 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:56.770176 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:56.770209 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:56.770241 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:56.770267 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:56.770308 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:56.770328 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:56.770349 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:56.770369 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:56.770405 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:56.770429 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:56.778126 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:56.778127 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:56.950289 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:57.344702 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.344722 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.344766 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.344789 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.344810 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.344831 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.344857 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.359301 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.359324 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.366399 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.371832 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.393369 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:57.396892 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.396920 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.414874 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.414896 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.423787 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:57.423791 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.423814 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.423834 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.423855 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:57.501627 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:57.501644 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:37:57.501647 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:37:57.501651 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:37:57.501659 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:37:57.501663 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:57.501665 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:57.501696 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:57.699022 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:57.699131 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:37:58.396098 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:58.396174 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:37:58.402335 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:58.402389 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:37:58.501747 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:58.501796 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:37:58.501809 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:58.501816 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:58.501812 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 321.226 < min (86400 seconds) 2014-05-19 18:37:58.501830 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:58.501833 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:58.501835 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:37:58.533620 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:58.533683 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:37:58.684981 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:58.685063 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:37:58.772391 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:58.772484 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:37:58.778236 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:58.778238 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:37:58.950384 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:37:59.281539 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:59.281685 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:37:59.320789 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:59.320877 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:37:59.344801 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.344824 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.344843 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.344862 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.344881 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.344900 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.344962 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.359389 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.359410 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.366497 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.371912 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.393464 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:37:59.396991 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.397014 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.414972 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.414997 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.423886 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:37:59.423889 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.423925 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.423944 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.423967 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:37:59.501924 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:37:59.501972 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:37:59.501985 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:37:59.501993 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:37:59.501989 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 322.226 < min (86400 seconds) 2014-05-19 18:37:59.502007 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:37:59.502010 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:37:59.502012 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:00.502078 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:00.502097 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:00.502101 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:00.502103 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:00.502134 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:00.599532 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:00.599634 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:00.702796 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:00.702896 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:00.778326 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:00.778328 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:00.950480 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:01.344879 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.344904 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.344942 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.344964 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.344985 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.345031 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.345058 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.359475 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.359500 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.366595 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.371996 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.393558 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:01.397089 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.397116 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.415053 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.415073 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.423995 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:01.423999 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.424036 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.424054 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.424108 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:01.502186 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:01.502207 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:01.502212 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:01.502214 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:01.502255 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:01.581980 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:01.582045 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:01.672990 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:01.673067 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:02.502296 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:02.502312 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:38:02.502316 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:38:02.502320 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:38:02.502328 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:02.502331 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:02.502333 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:02.502372 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:02.669936 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:02.669954 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:02.670397 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:02.670465 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:02.670502 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:02.670527 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:02.670553 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:02.670704 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:02.670765 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:02.670786 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:02.670806 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:02.670825 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:02.670866 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:02.670898 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:02.670943 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:02.670975 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:02.670997 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:02.671026 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:02.778426 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:02.778432 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:02.950572 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:03.344960 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.344980 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.345017 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.345039 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.345050 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.345111 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.345134 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.359557 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.359578 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.366693 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.372077 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.393655 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:03.397187 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.397210 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.415127 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.415151 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.424104 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:03.424107 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.424143 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.424162 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.424191 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:03.502418 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:03.502440 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:03.502444 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:03.502447 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:03.502487 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:03.603350 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:03.603436 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:03.696297 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:03.696360 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:04.434094 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:04.434179 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:04.482351 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:04.482420 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:04.502534 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:04.502585 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:04.502582 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:04.502595 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:04.502598 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 327.227 < min (86400 seconds) 2014-05-19 18:38:04.502616 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:04.502619 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:04.502621 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:04.585570 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:04.585622 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:04.621088 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:04.621152 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:04.778524 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:04.778527 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:04.950663 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:04.970480 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:04.970498 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:04.970951 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:04.971003 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:04.971035 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:04.971068 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:04.971105 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:04.971173 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:04.971203 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:04.971223 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:04.971241 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:04.971326 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:04.971353 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:04.971376 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:04.971401 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:04.971424 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:04.971446 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:04.971469 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:05.345049 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.345092 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.345114 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.345134 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.345156 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.345180 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.345207 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.359638 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.359656 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.366792 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.372161 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.393750 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:05.397285 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.397313 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.415207 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.415229 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.424218 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:05.424221 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.424259 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.424280 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.424331 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:05.502706 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:05.502727 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:05.502732 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:05.502734 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:05.502774 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:06.500164 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:06.500225 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:06.502795 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:06.502813 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:06.502818 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:06.502821 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:06.502849 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:06.734673 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:06.734741 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:06.778628 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:06.778630 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:06.886271 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:06.886330 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:06.950763 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:07.345147 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.345169 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.345181 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.345204 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.345229 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.345253 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.345269 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.359719 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.359743 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.366889 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.372255 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.393848 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:07.397386 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.397411 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.415288 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.415310 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.424328 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:07.424330 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.424368 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.424387 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.424441 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:07.502908 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:07.502925 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:38:07.502929 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:38:07.502933 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:38:07.502968 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:07.502980 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:07.502985 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:07.502983 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 330.227 < min (86400 seconds) 2014-05-19 18:38:07.503002 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:07.503005 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:07.503006 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:07.573529 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:07.573633 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:08.503041 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:08.503089 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:08.503102 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:08.503098 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:08.503105 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 331.227 < min (86400 seconds) 2014-05-19 18:38:08.503123 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:08.503126 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:08.503128 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:08.778713 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:08.778723 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:08.800679 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:08.800802 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:08.950836 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:08.996667 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:08.996736 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:09.345239 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.345261 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.345280 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.345300 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.345320 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.345333 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.345351 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.359809 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.359832 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.366987 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.372352 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.393943 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:09.397486 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.397511 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.415371 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.415391 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.424436 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:09.424440 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.424477 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.424495 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.424507 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:09.503197 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:09.503217 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:09.503222 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:09.503224 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:09.503265 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:09.503968 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:09.504058 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:09.670999 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:09.671017 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:09.671471 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:09.671558 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:09.671611 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:09.671636 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:09.671669 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:09.671738 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:09.671856 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:09.671878 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:09.671897 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:09.671921 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:09.671946 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:09.671968 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:09.671991 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:09.672013 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:09.672035 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:09.672060 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:09.874242 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:09.874326 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:09.921469 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:09.921528 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:10.382875 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:10.382935 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:10.503292 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:10.503314 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:10.503319 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:10.503321 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:10.503362 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:10.778798 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:10.778800 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:10.950926 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:11.345336 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.345355 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.345374 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.345395 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.345409 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.345419 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.345442 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.359885 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.359904 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.367084 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.372448 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.394040 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:11.397587 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.397615 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.415445 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.415468 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.424548 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:11.424551 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.424594 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.424620 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.424673 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:11.435132 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:11.435239 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:11.503392 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:11.503413 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:11.503418 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:11.503420 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:11.503443 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:11.586775 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:11.586835 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:11.804517 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:11.804605 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:12.503453 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:12.503469 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:38:12.503473 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:38:12.503477 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:38:12.503514 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:12.503526 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:12.503530 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:12.503529 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 335.228 < min (86400 seconds) 2014-05-19 18:38:12.503548 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:12.503551 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:12.503553 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:12.683338 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:12.683445 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:12.778899 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:12.778902 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:12.951021 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:13.345421 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.345447 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.345469 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.345491 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.345507 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.345526 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.345553 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.359961 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.359989 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.367167 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.372543 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.394120 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:13.397687 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.397713 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.415515 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.415546 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.424644 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:13.424649 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.424673 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.424727 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.424757 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:13.501356 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:13.501422 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:13.503622 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:13.503649 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:13.503672 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:13.503685 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:13.503688 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 336.228 < min (86400 seconds) 2014-05-19 18:38:13.503708 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:13.503711 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:13.503713 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:13.771560 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:13.771578 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:13.772047 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:13.772091 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:13.772116 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:13.772241 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:13.772312 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:13.772363 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:13.772400 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:13.772425 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:13.772450 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:13.772474 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:13.772499 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:13.772522 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:13.772548 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:13.772593 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:13.772619 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:13.772639 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:14.503797 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:14.503846 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:14.503859 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:14.503866 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:14.503862 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 337.228 < min (86400 seconds) 2014-05-19 18:38:14.503881 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:14.503883 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:14.503885 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:14.574685 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:14.574750 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:14.778992 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:14.778994 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:14.896944 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:14.897002 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:14.951102 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:15.345513 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.345548 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.345566 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.345583 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.345601 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.345632 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.345684 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.360037 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.360095 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.367250 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.372639 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.394199 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:15.397787 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.397811 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.415596 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.415618 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.424740 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:15.424745 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.424766 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.424793 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.424827 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:15.503970 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:15.503990 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:15.503995 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:15.503998 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:15.504038 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:15.535681 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:15.535744 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:15.687271 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:15.687553 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:15.821675 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:15.821738 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:16.504051 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:16.504104 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:16.504118 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:16.504120 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:16.504121 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 339.228 < min (86400 seconds) 2014-05-19 18:38:16.504139 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:16.504142 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:16.504144 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:16.504936 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:16.505017 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:16.779096 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:16.779098 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:16.951180 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:17.197315 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:17.197422 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:17.345605 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.345627 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.345649 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.345673 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.345693 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.345708 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.345754 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.360111 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.360174 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.367335 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.372720 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.383632 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:17.383836 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:17.394283 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:17.397889 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.397947 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.415677 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.415698 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.424842 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:17.424843 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.424879 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.424894 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.424910 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:17.504230 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:17.504247 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:38:17.504251 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:38:17.504254 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:38:17.504289 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:17.504301 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:17.504307 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:17.504304 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 340.228 < min (86400 seconds) 2014-05-19 18:38:17.504324 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:17.504327 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:17.504328 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:17.601920 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:17.602032 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:17.872065 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:17.872083 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:17.872663 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:17.872695 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:17.872721 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:17.872748 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:17.872790 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:17.872833 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:17.872872 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:17.872894 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:17.872933 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:17.872955 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:17.872979 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:17.873003 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:17.873027 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:17.873058 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:17.873082 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:17.873102 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:18.122061 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:18.122169 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:18.504410 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:18.504430 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:18.504435 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:18.504437 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:18.504467 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:18.675161 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:18.675257 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:18.779189 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:18.779192 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:18.951263 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:19.345675 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.345699 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.345731 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.345751 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.345768 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.345781 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.345814 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.360183 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.360289 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.367416 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.372816 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.394362 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:19.397988 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.398026 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.415776 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.415802 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.424951 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:19.424956 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.424982 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.425002 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.425023 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:19.504520 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:19.504542 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:19.504546 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:19.504549 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:19.504589 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:19.636202 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:19.636277 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:19.787866 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:19.788044 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:20.504626 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:20.504646 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:20.504651 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:20.504653 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:20.504683 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:20.605362 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:20.605430 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:20.772676 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:20.772695 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:20.773108 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:20.773225 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:20.773316 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:20.773367 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:20.773397 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:20.773540 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:20.773573 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:20.773593 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:20.773635 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:20.773662 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:20.773689 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:20.773713 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:20.773741 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:20.773764 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:20.773785 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:20.773809 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:20.779285 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:20.779287 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:20.951343 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:21.345763 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.345790 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.345811 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.345825 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.345845 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.345875 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.345895 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.360272 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.360361 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.367497 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.372913 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.394442 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:21.398086 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.398104 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.415875 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.415899 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.425047 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:21.425052 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.425074 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.425111 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.425125 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:21.484063 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:21.484140 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:21.504741 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:21.504762 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:21.504767 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:21.504769 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:21.504793 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:21.702451 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:21.702505 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:21.897530 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:21.897622 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:22.504851 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:22.504869 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:38:22.504874 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:38:22.504881 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:38:22.504892 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:22.504898 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:22.504902 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:22.504930 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:22.536738 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:22.536822 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:22.688444 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:22.688513 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:22.775702 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:22.775809 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:22.779380 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:22.779383 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:22.822326 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:22.822386 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:22.951436 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:23.345864 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.345890 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.345908 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.345922 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.345938 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.345950 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.345970 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.360357 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.360424 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.367589 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.373010 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.394526 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:23.398187 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.398213 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.415975 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.416001 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.425143 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:23.425147 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.425200 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.425223 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.425269 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:23.504987 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:23.505008 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:23.505013 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:23.505015 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:23.505056 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:24.505092 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:24.505141 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:24.505154 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:24.505149 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:24.505157 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 347.229 < min (86400 seconds) 2014-05-19 18:38:24.505176 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:24.505179 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:24.505180 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:24.603030 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:24.603101 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:24.705934 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:24.706058 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:24.779471 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:24.779474 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:24.951519 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:25.345944 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.345966 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.345993 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.346012 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.346027 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.346076 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.346131 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.360435 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.360489 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.367684 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.373107 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.394618 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:25.398287 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.398311 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.416075 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.416099 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.425250 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:25.425252 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.425281 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.425307 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.425350 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:25.505265 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:25.505285 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:25.505290 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:25.505293 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:25.505322 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:25.584612 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:25.584713 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:25.676126 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:25.676187 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:25.997884 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:25.997995 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:26.073207 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:26.073225 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:26.073810 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:26.073887 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:26.073978 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:26.074001 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:26.074020 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:26.074040 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:26.074065 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:26.074084 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:26.074103 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:26.074143 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:26.074171 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:26.074191 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:26.074211 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:26.074231 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:26.074251 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:26.074273 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:26.505374 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:26.505395 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:26.505400 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:26.505402 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:26.505443 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:26.779559 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:26.779564 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:26.922623 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:26.922682 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:26.951614 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:27.346007 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.346031 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.346061 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.346089 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.346112 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.346137 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.346203 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.360525 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.360557 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.367781 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.373202 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.394714 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:27.398388 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.398414 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.416175 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.416202 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.425356 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:27.425361 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.425390 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.425410 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.425428 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:27.505487 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:27.505504 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:38:27.505508 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:38:27.505511 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:38:27.505519 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:27.505523 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:27.505525 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:27.505564 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:27.606590 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:27.606666 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:27.837128 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:27.837216 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:27.989100 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:27.989165 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:28.373798 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:28.373816 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:28.374378 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:28.374418 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:28.374461 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:28.374505 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:28.374530 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:28.374553 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:28.374607 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:28.374657 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:28.374691 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:28.374716 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:28.374745 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:28.374768 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:28.374796 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:28.374818 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:28.374840 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:28.374862 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:28.484950 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:28.485036 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:28.505614 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:28.505662 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:28.505666 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:28.505676 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:28.505679 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 351.23 < min (86400 seconds) 2014-05-19 18:38:28.505697 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:28.505701 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:28.505703 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:28.779644 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:28.779646 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:28.951698 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:29.346049 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.346073 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.346134 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.346157 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.346178 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.346203 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.346267 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.360603 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.360667 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.367877 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.373298 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.394810 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:29.398488 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.398513 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.416273 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.416295 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.425452 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:29.425455 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.425493 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.425515 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.425562 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:29.505788 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:29.505809 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:29.505813 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:29.505816 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:29.505839 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:29.903721 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:29.903845 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:30.098196 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:30.098252 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:30.137665 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:30.137735 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:30.289730 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:30.289838 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:30.505902 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:30.505923 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:30.505927 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:30.505930 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:30.505960 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:30.779743 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:30.779746 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:30.951789 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:30.976619 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:30.976683 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:31.023009 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:31.023095 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:31.346124 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.346142 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.346239 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.346261 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.346275 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.346292 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.346330 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.360699 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.360771 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.367973 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.373397 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.394906 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:31.398591 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.398651 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.416373 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.416399 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.425557 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:31.425561 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.425576 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.425599 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.425630 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:31.505997 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:31.506019 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:31.506023 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:31.506026 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:31.506067 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:32.204171 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:32.204230 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:32.506096 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:32.506113 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:38:32.506117 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:38:32.506121 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:38:32.506156 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:32.506169 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:32.506174 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:32.506172 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 355.23 < min (86400 seconds) 2014-05-19 18:38:32.506191 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:32.506194 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:32.506196 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:32.779823 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:32.779845 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:32.907061 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:32.907142 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:32.951883 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:32.998686 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:32.998750 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:33.277143 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:33.277206 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:33.346221 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.346244 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.346317 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.346349 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.346370 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.346387 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.346447 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.360801 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.360853 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.368056 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.373495 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.395001 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:33.398690 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.398729 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.416471 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.416495 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.425658 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:33.425659 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.425695 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.425716 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.425765 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:33.506273 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:33.506295 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:33.506299 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:33.506302 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:33.506343 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:33.674456 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:33.674474 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:33.675015 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:33.675076 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:33.675111 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:33.675174 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:33.675207 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:33.675228 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:33.675253 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:33.675273 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:33.675293 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:33.675313 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:33.675334 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:33.675357 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:33.675380 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:33.675401 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:33.675422 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:33.675445 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:33.785140 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:33.785209 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:33.923364 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:33.923434 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:34.506368 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:34.506389 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:34.506394 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:34.506396 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:34.506437 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:34.775048 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:34.775067 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:34.775573 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:34.775648 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:34.775703 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:34.775758 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:34.775783 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:34.775807 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:34.775827 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:34.775855 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:34.775899 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:34.775947 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:34.775997 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:34.776025 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:34.776054 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:34.776080 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:34.776101 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:34.776123 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:34.779902 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:34.779921 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:34.951964 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:35.207676 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:35.207768 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:35.346292 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.346317 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.346418 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.346443 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.346461 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.346511 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.346541 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.360889 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.360917 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.368138 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.373592 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.395083 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:35.398789 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.398808 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.416570 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.416616 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.425766 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:35.425770 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.425792 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.425813 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.425842 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:35.438066 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:35.438192 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:35.506475 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:35.506496 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:35.506500 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:35.506503 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:35.506544 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:35.590286 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:35.590439 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:36.085578 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:36.085678 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:36.506570 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:36.506591 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:36.506595 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:36.506598 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:36.506638 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:36.538553 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:36.538667 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:36.690866 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:36.691001 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:36.779998 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:36.780000 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:36.952044 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:37.346361 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.346387 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.346479 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.346503 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.346528 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.346575 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.346612 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.360975 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.361029 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.368220 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.373689 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.395163 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:37.398891 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.398951 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.416688 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.416714 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.425864 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:37.425868 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.425906 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.425927 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.425976 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:37.504829 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:37.504925 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:37.506684 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:37.506701 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:38:37.506704 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:38:37.506708 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:38:37.506744 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:37.506757 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:37.506753 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:37.506760 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 360.231 < min (86400 seconds) 2014-05-19 18:38:37.506787 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:37.506790 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:37.506792 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:37.675640 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:37.675658 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:37.676139 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:37.676182 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:37.676340 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:37.676376 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:37.676419 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:37.676452 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:37.676503 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:37.676530 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:37.676556 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:37.676592 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:37.676619 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:37.676639 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:37.676665 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:37.676688 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:37.676709 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:37.676734 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:38.298958 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:38.299058 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:38.506879 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:38.506928 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:38.506941 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:38.506948 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:38.506944 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 361.231 < min (86400 seconds) 2014-05-19 18:38:38.506963 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:38.506966 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:38.506968 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:38.577571 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:38.577710 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:38.605298 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:38.605390 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:38.780094 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:38.780095 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:38.952132 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:39.223657 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:39.223747 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:39.346441 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.346466 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.346559 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.346578 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.346592 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.346638 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.346676 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.361069 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.361097 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.368300 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.373787 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.395245 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:39.398975 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.399031 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.416788 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.416815 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.425971 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:39.425976 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.426000 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.426019 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.426091 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:39.439079 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:39.439149 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:39.507044 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:39.507064 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:39.507069 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:39.507071 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:39.507112 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:39.591415 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:39.591540 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:39.678053 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:39.678103 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:40.507115 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:40.507135 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:40.507140 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:40.507142 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:40.507171 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:40.508298 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:40.508392 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:40.599395 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:40.599469 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:40.780193 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:40.780198 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:40.952224 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:41.346499 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.346517 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.346616 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.346641 7fd03d6b0700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.346658 7fd03f6b4700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.346701 7fd0436bc700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.346739 7fd03c6ae700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.361149 7fd0446be700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.361171 7fd03feb5700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.368384 7fd040eb7700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.373885 7fd0406b6700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.385908 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:41.385974 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:41.395328 7fd039ea9700 20 OSD::disk_tp worker waiting 2014-05-19 18:38:41.399059 7fd03aeab700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.399114 7fd03eeb3700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.416889 7fd042ebb700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.416915 7fd03deb1700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.426067 7fd04e104700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:41.426071 7fd03b6ac700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.426095 7fd0416b8700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.426118 7fd0426ba700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.426156 7fd03ceaf700 20 OSD::op_tp worker waiting 2014-05-19 18:38:41.505932 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:41.506057 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:41.507204 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:41.507264 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:41.507261 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes 2014-05-19 18:38:41.507279 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 2014-05-19 18:38:41.507284 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 364.231 < min (86400 seconds) 2014-05-19 18:38:41.507324 7fd04f907700 20 osd.0 21 sched_scrub done 2014-05-19 18:38:41.507330 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:41.507333 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:41.523927 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:41.524009 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:41.608783 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:41.608836 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:42.376238 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:42.376257 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) 2014-05-19 18:38:42.376738 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:42.376871 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:42.376918 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 2014-05-19 18:38:42.376987 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 2014-05-19 18:38:42.377015 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:42.377040 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:42.377095 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 2014-05-19 18:38:42.377139 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:42.377181 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:42.377209 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:42.377237 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:42.377260 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:42.377286 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:42.377309 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:42.377331 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:42.377355 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 2014-05-19 18:38:42.486410 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:42.486537 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:42.507410 7fd04f907700 5 osd.0 21 tick 2014-05-19 18:38:42.507426 7fd04f907700 7 osd.0 21 do_mon_report 2014-05-19 18:38:42.507430 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 2014-05-19 18:38:42.507434 7fd04f907700 20 osd.0 21 send_pg_stats 2014-05-19 18:38:42.507441 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off 2014-05-19 18:38:42.507445 7fd04f907700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:42.507447 7fd04f907700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:42.507461 7fd03a6aa700 20 OSD::recovery_tp worker waiting 2014-05-19 18:38:42.578548 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:42.578636 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:42.780287 7fd04e905700 20 KeyValueStore::op_tp worker waiting 2014-05-19 18:38:42.780289 7fd043ebd700 20 OSD::op_tp worker waiting 2014-05-19 18:38:42.899766 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:42.899847 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 2014-05-19 18:38:42.952317 7fd0396a8700 20 OSD::command_tp worker waiting 2014-05-19 18:38:43.001961 7fd02ef84700 10 osd.0 21 new session 0x2de9a40 con=0x61fba20 addr=172.16.96.219:0/1052969 2014-05-19 18:38:43.027060 7fd04aecb700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:43.027066 7fd04aecb700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:43.027069 7fd04aecb700 20 osd.0 21 _dispatch 0x66b8fc0 osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 2014-05-19 18:38:43.027086 7fd04aecb700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.006140, event: header_read, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 2014-05-19 18:38:43.027109 7fd04aecb700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.006146, event: throttled, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 2014-05-19 18:38:43.027139 7fd04aecb700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.026948, event: all_read, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 2014-05-19 18:38:43.027159 7fd04aecb700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.027056, event: dispatched, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 2014-05-19 18:38:43.027181 7fd04aecb700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.027181, event: waiting_for_osdmap, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 2014-05-19 18:38:43.027206 7fd04aecb700 15 osd.0 21 require_same_or_newer_map 21 (i am 21) 0x66b8fc0 2014-05-19 18:38:43.027213 7fd04aecb700 20 osd.0 21 _share_map_incoming client.4133 172.16.96.219:0/1052969 21 2014-05-19 18:38:43.027242 7fd04aecb700 15 osd.0 21 enqueue_op 0x5aaf2c0 prio 63 cost 4194304 latency 0.021101 osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 2014-05-19 18:38:43.027277 7fd04aecb700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:43.027282 7fd04aecb700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:43.027291 7fd03e6b2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active) 2014-05-19 18:38:43.027304 7fd03e6b2700 10 osd.0 21 dequeue_op 0x5aaf2c0 prio 63 cost 4194304 latency 0.021164 osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 pg pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] 2014-05-19 18:38:43.027331 7fd03e6b2700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.027331, event: reached_pg, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 2014-05-19 18:38:43.027355 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-05-19 18:38:43.027373 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] handle_message: 0x5aaf2c0 2014-05-19 18:38:43.027387 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_op osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 may_write -> write-ordered flags ack+ondisk+write 2014-05-19 18:38:43.027426 7fd03e6b2700 15 getattr 3.3f_head/d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 '_' 2014-05-19 18:38:43.027497 7fd03e6b2700 10 getattr got.size() is 0 2014-05-19 18:38:43.027503 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: no obc for soid d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 but can_create 2014-05-19 18:38:43.027522 7fd03e6b2700 15 getattr 3.3f_head/d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 'snapset' 2014-05-19 18:38:43.027549 7fd03e6b2700 10 getattr got.size() is 0 2014-05-19 18:38:43.027552 7fd03e6b2700 15 getattr 3.3f_head/d310c43f/rbd_data.10236b8b4567.0000000000000004/snapdir//3 'snapset' 2014-05-19 18:38:43.027576 7fd03e6b2700 10 getattr got.size() is 0 2014-05-19 18:38:43.027589 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] create_object_context 0x77ad080 d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 2014-05-19 18:38:43.027606 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] populate_obc_watchers d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 2014-05-19 18:38:43.027618 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 2014-05-19 18:38:43.027629 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: 0x77ad080 d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 rwstate(none n=0 w=0) oi: d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x2de8fc0 snapset: 0=[]:[] 2014-05-19 18:38:43.027649 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] find_object_context d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 @head oi=d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-05-19 18:38:43.027674 7fd03e6b2700 15 getattr 3.3f_head/d310c43f/rbd_data.10236b8b4567.0000000000000004/snapdir//3 '_' 2014-05-19 18:38:43.027705 7fd03e6b2700 10 getattr got.size() is 0 2014-05-19 18:38:43.027707 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: no obc for soid d310c43f/rbd_data.10236b8b4567.0000000000000004/snapdir//3 and !can_create 2014-05-19 18:38:43.027722 7fd03e6b2700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.027722, event: started, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 2014-05-19 18:38:43.027742 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] execute_ctx 0x6b0b600 2014-05-19 18:38:43.027767 7fd03e6b2700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.027767, event: started, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 2014-05-19 18:38:43.027782 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_op d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] ov 0'0 av 21'1 snapc 0=[] snapset 0=[]:[] 2014-05-19 18:38:43.027803 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 2014-05-19 18:38:43.027816 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-05-19 18:38:43.027837 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op write 0~4194304 2014-05-19 18:38:43.027863 7fd03e6b2700 15 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op_effects on session 0x2de9a40 2014-05-19 18:38:43.027881 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] make_writeable d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 snapset=0x2de8ff8 snapc=0=[] 2014-05-19 18:38:43.027894 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] setting DIRTY flag 2014-05-19 18:38:43.027904 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] make_writeable d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 done, snapset=0=[]:[]+head 2014-05-19 18:38:43.027917 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] finish_ctx d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 0x6b0b600 op modify 2014-05-19 18:38:43.027932 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] set mtime to 2014-05-19 12:37:55.600025 2014-05-19 18:38:43.027964 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] final snapset 0=[]:[]+head in d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 2014-05-19 18:38:43.027985 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] zeroing write result code 0 2014-05-19 18:38:43.027996 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] new_repop rep_tid 1 on osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 2014-05-19 18:38:43.028023 7fd03e6b2700 7 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] issue_repop rep_tid 1 o d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 2014-05-19 18:38:43.028048 7fd03e6b2700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.028048, event: waiting for subops from 41, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 2014-05-19 18:38:43.028082 7fd03e6b2700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 2014-05-19 18:38:43.028109 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] append_log log((0'0,0'0], crt=0'0) [21'1 (0'0) modify d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 by client.4133.0:104 2014-05-19 12:37:55.600025] 2014-05-19 18:38:43.028140 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] add_log_entry 21'1 (0'0) modify d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 by client.4133.0:104 2014-05-19 12:37:55.600025 2014-05-19 18:38:43.028168 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] append_log adding 1 keys 2014-05-19 18:38:43.028216 7fd03e6b2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 21'1, trimmed: 2014-05-19 18:38:43.028244 7fd03e6b2700 5 queue_transactions existing osr(3.3f 0x610ceb0)/0x610ceb0 2014-05-19 18:38:43.028249 7fd03e6b2700 10 op_submit_start 24886 2014-05-19 18:38:43.028251 7fd03e6b2700 5 queue_transactions (trailing journal) 24886 2014-05-19 18:38:43.028253 7fd03e6b2700 5 queue_op 0x40120c0 seq 24886 osr(3.3f 0x610ceb0) 4196755 bytes (queue has 1 ops and 4196755 bytes) 2014-05-19 18:38:43.028261 7fd03e6b2700 10 op_submit_finish 24886 2014-05-19 18:38:43.028271 7fd04e104700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x400de60 (1 active) 2014-05-19 18:38:43.028264 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] eval_repop repgather(0x30c4a80 21'1 rep_tid=1 committed?=0 applied?=0 lock=0 op=osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4) wants=ad 2014-05-19 18:38:43.028286 7fd04e104700 5 _do_op 0x40120c0 seq 24886 osr(3.3f 0x610ceb0)/0x610ceb0 start 2014-05-19 18:38:43.028290 7fd03e6b2700 10 osd.0 21 dequeue_op 0x5aaf2c0 finish 2014-05-19 18:38:43.028294 7fd04e104700 10 _do_transaction on 0x594c3c0 2014-05-19 18:38:43.028296 7fd03e6b2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active) 2014-05-19 18:38:43.028300 7fd03e6b2700 20 OSD::op_tp worker waiting 2014-05-19 18:38:43.028306 7fd04e104700 15 _omap_setkeys meta/51683abc/pglog_3.3f/0//-1 2014-05-19 18:38:43.028412 7fd04e104700 10 lookup_strip_header done cid meta oid 51683abc/pglog_3.3f/0//-1 2014-05-19 18:38:43.028452 7fd04e104700 10 _collection_setattr 3.3f_head 'info' len 1 2014-05-19 18:38:43.028544 7fd04e104700 10 lookup_strip_header done cid COLLECTIONS oid ecfe0533/3.3f_head/head//-1 2014-05-19 18:38:43.028586 7fd04e104700 10 _collection_setattr 3.3f_head 'info' len 1 = 0 2014-05-19 18:38:43.028603 7fd04e104700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-05-19 18:38:43.028685 7fd04e104700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-05-19 18:38:43.028728 7fd04e104700 15 _omap_rmkeys meta/51683abc/pglog_3.3f/0//-1 2014-05-19 18:38:43.028742 7fd04e104700 10 _omap_rmkeys meta/51683abc/pglog_3.3f/0//-1 r = 0 2014-05-19 18:38:43.028756 7fd04e104700 15 _omap_setkeys meta/51683abc/pglog_3.3f/0//-1 2014-05-19 18:38:43.028783 7fd04e104700 15 _touch 3.3f_head/d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 2014-05-19 18:38:43.028846 7fd04e104700 20 lookup_strip_header failed to get strip_header cid 3.3f_head oid d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 2014-05-19 18:38:43.028901 7fd04e104700 20 write_state seq is 8794 2014-05-19 18:38:43.028918 7fd04e104700 20 set_header setting 8793 cid 3.3f_head oid d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 parent seq 0 2014-05-19 18:38:43.028950 7fd04e104700 10 _touch 3.3f_head/d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 = 0 2014-05-19 18:38:43.028958 7fd04e104700 -1 bad op 2307 2014-05-19 18:38:43.032664 7fd04e104700 -1 os/KeyValueStore.cc: In function 'unsigned int KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)' thread 7fd04e104700 time 2014-05-19 18:38:43.028966 os/KeyValueStore.cc: 1457: FAILED assert(0) ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74) 1: (KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)+0x1d0) [0x9d8720] 2: (KeyValueStore::_do_transactions(std::list >&, unsigned long, ThreadPool::TPHandle*)+0x8e) [0x9daa0e] 3: (KeyValueStore::_do_op(KeyValueStore::OpSequencer*, ThreadPool::TPHandle&)+0x97) [0x9dab17] 4: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0xb5101a] 5: (ThreadPool::WorkThread::entry()+0x10) [0xb52270] 6: (()+0x7e9a) [0x7fd056078e9a] 7: (clone()+0x6d) [0x7fd054623ccd] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -10000> 2014-05-19 18:35:17.338577 7fd03f6b4700 20 OSD::op_tp worker waiting -9999> 2014-05-19 18:35:17.352051 7fd0446be700 20 OSD::op_tp worker waiting -9998> 2014-05-19 18:35:17.352074 7fd03feb5700 20 OSD::op_tp worker waiting -9997> 2014-05-19 18:35:17.359327 7fd040eb7700 20 OSD::op_tp worker waiting -9996> 2014-05-19 18:35:17.364390 7fd0406b6700 20 OSD::op_tp worker waiting -9995> 2014-05-19 18:35:17.379395 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 46 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:22.462703) v2 ==== 47+0+0 (961842693 0 0) 0x5667340 con 0x5b21600 -9994> 2014-05-19 18:35:17.379418 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 46 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:22.462703) v2 ==== 47+0+0 (961842693 0 0) 0x79628c0 con 0x5f5b760 -9993> 2014-05-19 18:35:17.379445 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:22.462703) v2 -- ?+0 0x669e540 con 0x5b21600 -9992> 2014-05-19 18:35:17.379480 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9991> 2014-05-19 18:35:17.379506 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:22.462703) v2 -- ?+0 0x5dffdc0 con 0x5f5b760 -9990> 2014-05-19 18:35:17.379584 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9989> 2014-05-19 18:35:17.385978 7fd039ea9700 20 OSD::disk_tp worker waiting -9988> 2014-05-19 18:35:17.389374 7fd03aeab700 20 OSD::op_tp worker waiting -9987> 2014-05-19 18:35:17.389397 7fd03eeb3700 20 OSD::op_tp worker waiting -9986> 2014-05-19 18:35:17.407411 7fd042ebb700 20 OSD::op_tp worker waiting -9985> 2014-05-19 18:35:17.407438 7fd03deb1700 20 OSD::op_tp worker waiting -9984> 2014-05-19 18:35:17.415751 7fd04e104700 20 KeyValueStore::op_tp worker waiting -9983> 2014-05-19 18:35:17.415753 7fd0426ba700 20 OSD::op_tp worker waiting -9982> 2014-05-19 18:35:17.415791 7fd03b6ac700 20 OSD::op_tp worker waiting -9981> 2014-05-19 18:35:17.415814 7fd03ceaf700 20 OSD::op_tp worker waiting -9980> 2014-05-19 18:35:17.415833 7fd0416b8700 20 OSD::op_tp worker waiting -9979> 2014-05-19 18:35:17.447048 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 47 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:20.503055) v2 ==== 47+0+0 (2233404274 0 0) 0x79e68c0 con 0x4cb8dc0 -9978> 2014-05-19 18:35:17.447087 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:20.503055) v2 -- ?+0 0x5667340 con 0x4cb8dc0 -9977> 2014-05-19 18:35:17.447068 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 47 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:20.503055) v2 ==== 47+0+0 (2233404274 0 0) 0x691b6c0 con 0x4cb9340 -9976> 2014-05-19 18:35:17.447162 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9975> 2014-05-19 18:35:17.447196 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:20.503055) v2 -- ?+0 0x79628c0 con 0x4cb9340 -9974> 2014-05-19 18:35:17.447241 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9973> 2014-05-19 18:35:17.480499 7fd04f907700 5 osd.0 21 tick -9972> 2014-05-19 18:35:17.480516 7fd04f907700 7 osd.0 21 do_mon_report -9971> 2014-05-19 18:35:17.480520 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -9970> 2014-05-19 18:35:17.480523 7fd04f907700 20 osd.0 21 send_pg_stats -9969> 2014-05-19 18:35:17.480558 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.23 < max 0.5 = yes -9968> 2014-05-19 18:35:17.480570 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -9967> 2014-05-19 18:35:17.480603 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9966> 2014-05-19 18:35:17.480573 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 160.205 < min (86400 seconds) -9965> 2014-05-19 18:35:17.480618 7fd04f907700 20 osd.0 21 sched_scrub done -9964> 2014-05-19 18:35:17.480622 7fd04f907700 10 osd.0 21 do_waiters -- start -9963> 2014-05-19 18:35:17.480623 7fd04f907700 10 osd.0 21 do_waiters -- finish -9962> 2014-05-19 18:35:17.840861 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9961> 2014-05-19 18:35:17.840879 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9960> 2014-05-19 18:35:17.840887 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x7930540 con 0x3622580 -9959> 2014-05-19 18:35:17.840914 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x6c4d6c0 con 0x3622840 -9958> 2014-05-19 18:35:17.840951 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x6113880 con 0x3622b00 -9957> 2014-05-19 18:35:17.840965 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x6710fc0 con 0x3622dc0 -9956> 2014-05-19 18:35:17.840979 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x692cc40 con 0x30594a0 -9955> 2014-05-19 18:35:17.841006 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x5519dc0 con 0x3622f20 -9954> 2014-05-19 18:35:17.841041 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x6504700 con 0x5e7ef20 -9953> 2014-05-19 18:35:17.841071 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x51a6540 con 0x5e7f1e0 -9952> 2014-05-19 18:35:17.841098 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x64afc00 con 0x5e7e2c0 -9951> 2014-05-19 18:35:17.841126 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x8388700 con 0x5d7bb80 -9950> 2014-05-19 18:35:17.841151 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x6542c40 con 0x5e7ec60 -9949> 2014-05-19 18:35:17.841236 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x7746c40 con 0x3622580 -9948> 2014-05-19 18:35:17.841261 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x64ae8c0 con 0x531b760 -9947> 2014-05-19 18:35:17.841291 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x60096c0 con 0x53bba20 -9946> 2014-05-19 18:35:17.841316 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x5f0b180 con 0x51cac60 -9945> 2014-05-19 18:35:17.841341 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x5ccc1c0 con 0x5bd66e0 -9944> 2014-05-19 18:35:17.841366 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:17.840885) v2 -- ?+0 0x79988c0 con 0x5bd6840 -9943> 2014-05-19 18:35:17.841451 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9942> 2014-05-19 18:35:17.841475 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x6fe61c0 con 0x3622dc0 -9941> 2014-05-19 18:35:17.841520 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9940> 2014-05-19 18:35:17.841531 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x7733a40 con 0x5e7ef20 -9939> 2014-05-19 18:35:17.841548 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9938> 2014-05-19 18:35:17.841556 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x6543180 con 0x5d7bb80 -9937> 2014-05-19 18:35:17.841607 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9936> 2014-05-19 18:35:17.841622 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x6b70540 con 0x30594a0 -9935> 2014-05-19 18:35:17.841648 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9934> 2014-05-19 18:35:17.841658 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x3511a40 con 0x3622f20 -9933> 2014-05-19 18:35:17.841673 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9932> 2014-05-19 18:35:17.841683 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x682f340 con 0x3622840 -9931> 2014-05-19 18:35:17.841698 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9930> 2014-05-19 18:35:17.841703 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x83881c0 con 0x3622b00 -9929> 2014-05-19 18:35:17.841717 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9928> 2014-05-19 18:35:17.841756 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x6c4d500 con 0x5e7f1e0 -9927> 2014-05-19 18:35:17.841784 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9926> 2014-05-19 18:35:17.841795 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x7a5f180 con 0x5e7e2c0 -9925> 2014-05-19 18:35:17.841814 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9924> 2014-05-19 18:35:17.841827 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x692dc00 con 0x5e7ec60 -9923> 2014-05-19 18:35:17.841847 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9922> 2014-05-19 18:35:17.841852 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x66f81c0 con 0x531b760 -9921> 2014-05-19 18:35:17.841867 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9920> 2014-05-19 18:35:17.841873 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 45 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x70a8540 con 0x53bba20 -9919> 2014-05-19 18:35:17.841888 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9918> 2014-05-19 18:35:17.841893 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 45 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x5f7a8c0 con 0x51cac60 -9917> 2014-05-19 18:35:17.841907 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9916> 2014-05-19 18:35:17.841915 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 44 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x76acfc0 con 0x5bd66e0 -9915> 2014-05-19 18:35:17.841929 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9914> 2014-05-19 18:35:17.841937 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 44 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:17.840885) v2 ==== 47+0+0 (1273084867 0 0) 0x5497dc0 con 0x5bd6840 -9913> 2014-05-19 18:35:17.841954 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9912> 2014-05-19 18:35:18.304482 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 43 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:23.387870) v2 ==== 47+0+0 (1893913601 0 0) 0x669ee00 con 0x30946e0 -9911> 2014-05-19 18:35:18.304529 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.387870) v2 -- ?+0 0x79e68c0 con 0x30946e0 -9910> 2014-05-19 18:35:18.304529 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 43 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:23.387870) v2 ==== 47+0+0 (1893913601 0 0) 0x5dff180 con 0x3094160 -9909> 2014-05-19 18:35:18.304573 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9908> 2014-05-19 18:35:18.304619 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.387870) v2 -- ?+0 0x691b6c0 con 0x3094160 -9907> 2014-05-19 18:35:18.304664 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9906> 2014-05-19 18:35:18.480697 7fd04f907700 5 osd.0 21 tick -9905> 2014-05-19 18:35:18.480745 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.21 < max 0.5 = yes -9904> 2014-05-19 18:35:18.480758 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -9903> 2014-05-19 18:35:18.480766 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9902> 2014-05-19 18:35:18.480761 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 161.205 < min (86400 seconds) -9901> 2014-05-19 18:35:18.480780 7fd04f907700 20 osd.0 21 sched_scrub done -9900> 2014-05-19 18:35:18.480782 7fd04f907700 10 osd.0 21 do_waiters -- start -9899> 2014-05-19 18:35:18.480784 7fd04f907700 10 osd.0 21 do_waiters -- finish -9898> 2014-05-19 18:35:18.771080 7fd043ebd700 20 OSD::op_tp worker waiting -9897> 2014-05-19 18:35:18.771408 7fd04e905700 20 KeyValueStore::op_tp worker waiting -9896> 2014-05-19 18:35:18.943423 7fd0396a8700 20 OSD::command_tp worker waiting -9895> 2014-05-19 18:35:19.337734 7fd041eb9700 20 OSD::op_tp worker waiting -9894> 2014-05-19 18:35:19.337778 7fd03e6b2700 20 OSD::op_tp worker waiting -9893> 2014-05-19 18:35:19.337804 7fd03c6ae700 20 OSD::op_tp worker waiting -9892> 2014-05-19 18:35:19.338502 7fd03d6b0700 20 OSD::op_tp worker waiting -9891> 2014-05-19 18:35:19.338589 7fd0436bc700 20 OSD::op_tp worker waiting -9890> 2014-05-19 18:35:19.338607 7fd03bead700 20 OSD::op_tp worker waiting -9889> 2014-05-19 18:35:19.338635 7fd03f6b4700 20 OSD::op_tp worker waiting -9888> 2014-05-19 18:35:19.352150 7fd0446be700 20 OSD::op_tp worker waiting -9887> 2014-05-19 18:35:19.352176 7fd03feb5700 20 OSD::op_tp worker waiting -9886> 2014-05-19 18:35:19.359419 7fd040eb7700 20 OSD::op_tp worker waiting -9885> 2014-05-19 18:35:19.364470 7fd0406b6700 20 OSD::op_tp worker waiting -9884> 2014-05-19 18:35:19.386073 7fd039ea9700 20 OSD::disk_tp worker waiting -9883> 2014-05-19 18:35:19.389477 7fd03aeab700 20 OSD::op_tp worker waiting -9882> 2014-05-19 18:35:19.389502 7fd03eeb3700 20 OSD::op_tp worker waiting -9881> 2014-05-19 18:35:19.407510 7fd042ebb700 20 OSD::op_tp worker waiting -9880> 2014-05-19 18:35:19.407531 7fd03deb1700 20 OSD::op_tp worker waiting -9879> 2014-05-19 18:35:19.415858 7fd04e104700 20 KeyValueStore::op_tp worker waiting -9878> 2014-05-19 18:35:19.415861 7fd0426ba700 20 OSD::op_tp worker waiting -9877> 2014-05-19 18:35:19.415897 7fd03b6ac700 20 OSD::op_tp worker waiting -9876> 2014-05-19 18:35:19.415921 7fd0416b8700 20 OSD::op_tp worker waiting -9875> 2014-05-19 18:35:19.415940 7fd03ceaf700 20 OSD::op_tp worker waiting -9874> 2014-05-19 18:35:19.480862 7fd04f907700 5 osd.0 21 tick -9873> 2014-05-19 18:35:19.480910 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.21 < max 0.5 = yes -9872> 2014-05-19 18:35:19.480923 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -9871> 2014-05-19 18:35:19.480929 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9870> 2014-05-19 18:35:19.480927 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 162.205 < min (86400 seconds) -9869> 2014-05-19 18:35:19.480945 7fd04f907700 20 osd.0 21 sched_scrub done -9868> 2014-05-19 18:35:19.480948 7fd04f907700 10 osd.0 21 do_waiters -- start -9867> 2014-05-19 18:35:19.480950 7fd04f907700 10 osd.0 21 do_waiters -- finish -9866> 2014-05-19 18:35:19.608912 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:22.664902) v2 ==== 47+0+0 (3172305129 0 0) 0x5666c40 con 0x61ae420 -9865> 2014-05-19 18:35:19.608918 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:22.664902) v2 ==== 47+0+0 (3172305129 0 0) 0x2ee16c0 con 0x618fe40 -9864> 2014-05-19 18:35:19.608955 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:22.664902) v2 -- ?+0 0x669ee00 con 0x61ae420 -9863> 2014-05-19 18:35:19.608991 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9862> 2014-05-19 18:35:19.609050 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:22.664902) v2 -- ?+0 0x5dff180 con 0x618fe40 -9861> 2014-05-19 18:35:19.609102 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9860> 2014-05-19 18:35:19.757282 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 46 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:19.757089) v2 ==== 47+0+0 (3079886477 0 0) 0x7931a40 con 0x4cb8160 -9859> 2014-05-19 18:35:19.757314 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 46 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:19.757089) v2 ==== 47+0+0 (3079886477 0 0) 0x55f4380 con 0x4cb82c0 -9858> 2014-05-19 18:35:19.757335 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:19.757089) v2 -- ?+0 0x5666c40 con 0x4cb8160 -9857> 2014-05-19 18:35:19.757385 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9856> 2014-05-19 18:35:19.757407 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:19.757089) v2 -- ?+0 0x2ee16c0 con 0x4cb82c0 -9855> 2014-05-19 18:35:19.757450 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9854> 2014-05-19 18:35:19.906250 7fd0456c0700 10 monclient: tick -9853> 2014-05-19 18:35:19.906263 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:34:49.906261) -9852> 2014-05-19 18:35:19.906279 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:35:19.906279; renew after: 2014-05-19 18:35:39.994233) -- no -9851> 2014-05-19 18:35:20.481034 7fd04f907700 5 osd.0 21 tick -9850> 2014-05-19 18:35:20.481055 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -9849> 2014-05-19 18:35:20.481059 7fd04f907700 10 osd.0 21 do_waiters -- start -9848> 2014-05-19 18:35:20.481062 7fd04f907700 10 osd.0 21 do_waiters -- finish -9847> 2014-05-19 18:35:20.481103 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9846> 2014-05-19 18:35:20.576773 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 46 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:23.632799) v2 ==== 47+0+0 (4180279299 0 0) 0x6113500 con 0x61ae160 -9845> 2014-05-19 18:35:20.576780 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 46 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:23.632799) v2 ==== 47+0+0 (4180279299 0 0) 0x60f6000 con 0x61b0420 -9844> 2014-05-19 18:35:20.576833 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.632799) v2 -- ?+0 0x7931a40 con 0x61ae160 -9843> 2014-05-19 18:35:20.576872 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9842> 2014-05-19 18:35:20.576935 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.632799) v2 -- ?+0 0x55f4380 con 0x61b0420 -9841> 2014-05-19 18:35:20.576967 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9840> 2014-05-19 18:35:20.771179 7fd043ebd700 20 OSD::op_tp worker waiting -9839> 2014-05-19 18:35:20.771471 7fd04e905700 20 KeyValueStore::op_tp worker waiting -9838> 2014-05-19 18:35:20.943515 7fd0396a8700 20 OSD::command_tp worker waiting -9837> 2014-05-19 18:35:20.972705 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 43 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:20.972540) v2 ==== 47+0+0 (3249844808 0 0) 0x77e8000 con 0x4cb86e0 -9836> 2014-05-19 18:35:20.972758 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:20.972540) v2 -- ?+0 0x6113500 con 0x4cb86e0 -9835> 2014-05-19 18:35:20.972752 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 43 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:20.972540) v2 ==== 47+0+0 (3249844808 0 0) 0x5667c00 con 0x4cb8420 -9834> 2014-05-19 18:35:20.972793 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9833> 2014-05-19 18:35:20.972820 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:20.972540) v2 -- ?+0 0x60f6000 con 0x4cb8420 -9832> 2014-05-19 18:35:20.972887 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9831> 2014-05-19 18:35:21.337823 7fd041eb9700 20 OSD::op_tp worker waiting -9830> 2014-05-19 18:35:21.337851 7fd03e6b2700 20 OSD::op_tp worker waiting -9829> 2014-05-19 18:35:21.337873 7fd03c6ae700 20 OSD::op_tp worker waiting -9828> 2014-05-19 18:35:21.338582 7fd03d6b0700 20 OSD::op_tp worker waiting -9827> 2014-05-19 18:35:21.338671 7fd0436bc700 20 OSD::op_tp worker waiting -9826> 2014-05-19 18:35:21.338698 7fd03bead700 20 OSD::op_tp worker waiting -9825> 2014-05-19 18:35:21.338715 7fd03f6b4700 20 OSD::op_tp worker waiting -9824> 2014-05-19 18:35:21.352240 7fd0446be700 20 OSD::op_tp worker waiting -9823> 2014-05-19 18:35:21.352266 7fd03feb5700 20 OSD::op_tp worker waiting -9822> 2014-05-19 18:35:21.359517 7fd040eb7700 20 OSD::op_tp worker waiting -9821> 2014-05-19 18:35:21.364551 7fd0406b6700 20 OSD::op_tp worker waiting -9820> 2014-05-19 18:35:21.386168 7fd039ea9700 20 OSD::disk_tp worker waiting -9819> 2014-05-19 18:35:21.389578 7fd03aeab700 20 OSD::op_tp worker waiting -9818> 2014-05-19 18:35:21.389603 7fd03eeb3700 20 OSD::op_tp worker waiting -9817> 2014-05-19 18:35:21.407609 7fd042ebb700 20 OSD::op_tp worker waiting -9816> 2014-05-19 18:35:21.407635 7fd03deb1700 20 OSD::op_tp worker waiting -9815> 2014-05-19 18:35:21.415964 7fd04e104700 20 KeyValueStore::op_tp worker waiting -9814> 2014-05-19 18:35:21.415966 7fd0426ba700 20 OSD::op_tp worker waiting -9813> 2014-05-19 18:35:21.415999 7fd0416b8700 20 OSD::op_tp worker waiting -9812> 2014-05-19 18:35:21.416028 7fd03ceaf700 20 OSD::op_tp worker waiting -9811> 2014-05-19 18:35:21.416050 7fd03b6ac700 20 OSD::op_tp worker waiting -9810> 2014-05-19 18:35:21.462641 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:26.546138) v2 ==== 47+0+0 (4272952460 0 0) 0x55188c0 con 0x31e1600 -9809> 2014-05-19 18:35:21.462676 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:26.546138) v2 ==== 47+0+0 (4272952460 0 0) 0x5c05500 con 0x61b0160 -9808> 2014-05-19 18:35:21.462685 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:26.546138) v2 -- ?+0 0x77e8000 con 0x31e1600 -9807> 2014-05-19 18:35:21.462749 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9806> 2014-05-19 18:35:21.462771 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:26.546138) v2 -- ?+0 0x5667c00 con 0x61b0160 -9805> 2014-05-19 18:35:21.462860 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9804> 2014-05-19 18:35:21.479746 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 47 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:26.563224) v2 ==== 47+0+0 (3071862950 0 0) 0x772d340 con 0x5b21600 -9803> 2014-05-19 18:35:21.479792 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:26.563224) v2 -- ?+0 0x55188c0 con 0x5b21600 -9802> 2014-05-19 18:35:21.479820 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9801> 2014-05-19 18:35:21.479906 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 47 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:26.563224) v2 ==== 47+0+0 (3071862950 0 0) 0x5c05a40 con 0x5f5b760 -9800> 2014-05-19 18:35:21.479938 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:26.563224) v2 -- ?+0 0x5c05500 con 0x5f5b760 -9799> 2014-05-19 18:35:21.479960 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9798> 2014-05-19 18:35:21.481149 7fd04f907700 5 osd.0 21 tick -9797> 2014-05-19 18:35:21.481197 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.21 < max 0.5 = yes -9796> 2014-05-19 18:35:21.481210 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -9795> 2014-05-19 18:35:21.481206 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9794> 2014-05-19 18:35:21.481213 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 164.205 < min (86400 seconds) -9793> 2014-05-19 18:35:21.481232 7fd04f907700 20 osd.0 21 sched_scrub done -9792> 2014-05-19 18:35:21.481235 7fd04f907700 10 osd.0 21 do_waiters -- start -9791> 2014-05-19 18:35:21.481236 7fd04f907700 10 osd.0 21 do_waiters -- finish -9790> 2014-05-19 18:35:22.404812 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 44 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:27.488338) v2 ==== 47+0+0 (666233781 0 0) 0x5dff880 con 0x3094160 -9789> 2014-05-19 18:35:22.404861 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.488338) v2 -- ?+0 0x5c05a40 con 0x3094160 -9788> 2014-05-19 18:35:22.404908 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9787> 2014-05-19 18:35:22.404820 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 44 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:27.488338) v2 ==== 47+0+0 (666233781 0 0) 0x669f340 con 0x30946e0 -9786> 2014-05-19 18:35:22.404969 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.488338) v2 -- ?+0 0x772d340 con 0x30946e0 -9785> 2014-05-19 18:35:22.404998 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9784> 2014-05-19 18:35:22.481320 7fd04f907700 5 osd.0 21 tick -9783> 2014-05-19 18:35:22.481337 7fd04f907700 7 osd.0 21 do_mon_report -9782> 2014-05-19 18:35:22.481341 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -9781> 2014-05-19 18:35:22.481345 7fd04f907700 20 osd.0 21 send_pg_stats -9780> 2014-05-19 18:35:22.481352 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -9779> 2014-05-19 18:35:22.481356 7fd04f907700 10 osd.0 21 do_waiters -- start -9778> 2014-05-19 18:35:22.481358 7fd04f907700 10 osd.0 21 do_waiters -- finish -9777> 2014-05-19 18:35:22.481371 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9776> 2014-05-19 18:35:22.747460 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 48 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:25.803511) v2 ==== 47+0+0 (4131755180 0 0) 0x79e6fc0 con 0x4cb8dc0 -9775> 2014-05-19 18:35:22.747483 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 48 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:25.803511) v2 ==== 47+0+0 (4131755180 0 0) 0x691bdc0 con 0x4cb9340 -9774> 2014-05-19 18:35:22.747504 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.803511) v2 -- ?+0 0x669f340 con 0x4cb8dc0 -9773> 2014-05-19 18:35:22.747560 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9772> 2014-05-19 18:35:22.747586 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.803511) v2 -- ?+0 0x5dff880 con 0x4cb9340 -9771> 2014-05-19 18:35:22.747673 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9770> 2014-05-19 18:35:22.771277 7fd043ebd700 20 OSD::op_tp worker waiting -9769> 2014-05-19 18:35:22.771548 7fd04e905700 20 KeyValueStore::op_tp worker waiting -9768> 2014-05-19 18:35:22.943599 7fd0396a8700 20 OSD::command_tp worker waiting -9767> 2014-05-19 18:35:23.141504 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9766> 2014-05-19 18:35:23.141522 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9765> 2014-05-19 18:35:23.141530 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x5f06000 con 0x3622580 -9764> 2014-05-19 18:35:23.141555 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x798fc00 con 0x3622840 -9763> 2014-05-19 18:35:23.141593 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x692d180 con 0x3622b00 -9762> 2014-05-19 18:35:23.141608 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x5c468c0 con 0x3622dc0 -9761> 2014-05-19 18:35:23.141621 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x6d616c0 con 0x30594a0 -9760> 2014-05-19 18:35:23.141646 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x3511180 con 0x3622f20 -9759> 2014-05-19 18:35:23.141677 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x2f4fdc0 con 0x5e7ef20 -9758> 2014-05-19 18:35:23.141703 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x662bdc0 con 0x5e7f1e0 -9757> 2014-05-19 18:35:23.141800 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x7747180 con 0x5e7e2c0 -9756> 2014-05-19 18:35:23.141849 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x73c2e00 con 0x5d7bb80 -9755> 2014-05-19 18:35:23.141876 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x798fdc0 con 0x5e7ec60 -9754> 2014-05-19 18:35:23.141903 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x5ccc8c0 con 0x531b760 -9753> 2014-05-19 18:35:23.141895 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x77476c0 con 0x3622580 -9752> 2014-05-19 18:35:23.141946 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x682e540 con 0x53bba20 -9751> 2014-05-19 18:35:23.141972 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x3511340 con 0x51cac60 -9750> 2014-05-19 18:35:23.141999 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x2f4efc0 con 0x5bd66e0 -9749> 2014-05-19 18:35:23.142015 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:23.141528) v2 -- ?+0 0x6009340 con 0x5bd6840 -9748> 2014-05-19 18:35:23.142054 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9747> 2014-05-19 18:35:23.142073 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x7999180 con 0x3622b00 -9746> 2014-05-19 18:35:23.142097 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9745> 2014-05-19 18:35:23.142108 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x6fe7a40 con 0x3622dc0 -9744> 2014-05-19 18:35:23.142124 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9743> 2014-05-19 18:35:23.142129 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x6b716c0 con 0x30594a0 -9742> 2014-05-19 18:35:23.142166 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9741> 2014-05-19 18:35:23.142173 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x6efefc0 con 0x3622840 -9740> 2014-05-19 18:35:23.142188 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9739> 2014-05-19 18:35:23.142199 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x7733dc0 con 0x5e7ef20 -9738> 2014-05-19 18:35:23.142214 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9737> 2014-05-19 18:35:23.142220 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x7a5e1c0 con 0x5e7e2c0 -9736> 2014-05-19 18:35:23.142247 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9735> 2014-05-19 18:35:23.142267 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x3511500 con 0x3622f20 -9734> 2014-05-19 18:35:23.142322 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9733> 2014-05-19 18:35:23.142335 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x6009dc0 con 0x5d7bb80 -9732> 2014-05-19 18:35:23.142360 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9731> 2014-05-19 18:35:23.142371 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x692da40 con 0x5e7ec60 -9730> 2014-05-19 18:35:23.142398 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9729> 2014-05-19 18:35:23.142414 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x66f9a40 con 0x531b760 -9728> 2014-05-19 18:35:23.142438 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9727> 2014-05-19 18:35:23.142445 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x70a81c0 con 0x53bba20 -9726> 2014-05-19 18:35:23.142460 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9725> 2014-05-19 18:35:23.142474 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x60f6e00 con 0x51cac60 -9724> 2014-05-19 18:35:23.142492 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9723> 2014-05-19 18:35:23.142499 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 45 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x76addc0 con 0x5bd66e0 -9722> 2014-05-19 18:35:23.142514 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9721> 2014-05-19 18:35:23.142522 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 45 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x5497340 con 0x5bd6840 -9720> 2014-05-19 18:35:23.142536 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9719> 2014-05-19 18:35:23.142543 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:23.141528) v2 ==== 47+0+0 (1551325813 0 0) 0x6c4da40 con 0x5e7f1e0 -9718> 2014-05-19 18:35:23.142558 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9717> 2014-05-19 18:35:23.337914 7fd041eb9700 20 OSD::op_tp worker waiting -9716> 2014-05-19 18:35:23.337938 7fd03e6b2700 20 OSD::op_tp worker waiting -9715> 2014-05-19 18:35:23.337962 7fd03c6ae700 20 OSD::op_tp worker waiting -9714> 2014-05-19 18:35:23.338664 7fd03d6b0700 20 OSD::op_tp worker waiting -9713> 2014-05-19 18:35:23.338755 7fd0436bc700 20 OSD::op_tp worker waiting -9712> 2014-05-19 18:35:23.338776 7fd03bead700 20 OSD::op_tp worker waiting -9711> 2014-05-19 18:35:23.338802 7fd03f6b4700 20 OSD::op_tp worker waiting -9710> 2014-05-19 18:35:23.352337 7fd0446be700 20 OSD::op_tp worker waiting -9709> 2014-05-19 18:35:23.352361 7fd03feb5700 20 OSD::op_tp worker waiting -9708> 2014-05-19 18:35:23.359609 7fd040eb7700 20 OSD::op_tp worker waiting -9707> 2014-05-19 18:35:23.364634 7fd0406b6700 20 OSD::op_tp worker waiting -9706> 2014-05-19 18:35:23.386263 7fd039ea9700 20 OSD::disk_tp worker waiting -9705> 2014-05-19 18:35:23.389678 7fd03aeab700 20 OSD::op_tp worker waiting -9704> 2014-05-19 18:35:23.389706 7fd03eeb3700 20 OSD::op_tp worker waiting -9703> 2014-05-19 18:35:23.407710 7fd042ebb700 20 OSD::op_tp worker waiting -9702> 2014-05-19 18:35:23.407737 7fd03deb1700 20 OSD::op_tp worker waiting -9701> 2014-05-19 18:35:23.416070 7fd04e104700 20 KeyValueStore::op_tp worker waiting -9700> 2014-05-19 18:35:23.416073 7fd0426ba700 20 OSD::op_tp worker waiting -9699> 2014-05-19 18:35:23.416104 7fd03ceaf700 20 OSD::op_tp worker waiting -9698> 2014-05-19 18:35:23.416127 7fd0416b8700 20 OSD::op_tp worker waiting -9697> 2014-05-19 18:35:23.416141 7fd03b6ac700 20 OSD::op_tp worker waiting -9696> 2014-05-19 18:35:23.481445 7fd04f907700 5 osd.0 21 tick -9695> 2014-05-19 18:35:23.481494 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.2 < max 0.5 = yes -9694> 2014-05-19 18:35:23.481507 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -9693> 2014-05-19 18:35:23.481513 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9692> 2014-05-19 18:35:23.481510 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 166.206 < min (86400 seconds) -9691> 2014-05-19 18:35:23.481528 7fd04f907700 20 osd.0 21 sched_scrub done -9690> 2014-05-19 18:35:23.481531 7fd04f907700 10 osd.0 21 do_waiters -- start -9689> 2014-05-19 18:35:23.481533 7fd04f907700 10 osd.0 21 do_waiters -- finish -9688> 2014-05-19 18:35:24.481621 7fd04f907700 5 osd.0 21 tick -9687> 2014-05-19 18:35:24.481642 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -9686> 2014-05-19 18:35:24.481646 7fd04f907700 10 osd.0 21 do_waiters -- start -9685> 2014-05-19 18:35:24.481648 7fd04f907700 10 osd.0 21 do_waiters -- finish -9684> 2014-05-19 18:35:24.481690 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9683> 2014-05-19 18:35:24.677517 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 47 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:27.733556) v2 ==== 47+0+0 (4235931991 0 0) 0x64ae1c0 con 0x61ae160 -9682> 2014-05-19 18:35:24.677558 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.733556) v2 -- ?+0 0x79e6fc0 con 0x61ae160 -9681> 2014-05-19 18:35:24.677588 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 47 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:27.733556) v2 ==== 47+0+0 (4235931991 0 0) 0x60f68c0 con 0x61b0420 -9680> 2014-05-19 18:35:24.677632 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9679> 2014-05-19 18:35:24.677658 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.733556) v2 -- ?+0 0x691bdc0 con 0x61b0420 -9678> 2014-05-19 18:35:24.677691 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9677> 2014-05-19 18:35:24.771377 7fd043ebd700 20 OSD::op_tp worker waiting -9676> 2014-05-19 18:35:24.771618 7fd04e905700 20 KeyValueStore::op_tp worker waiting -9675> 2014-05-19 18:35:24.909400 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:27.965468) v2 ==== 47+0+0 (1400428620 0 0) 0x75b0e00 con 0x618fe40 -9674> 2014-05-19 18:35:24.909409 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:27.965468) v2 ==== 47+0+0 (1400428620 0 0) 0x5fc3880 con 0x61ae420 -9673> 2014-05-19 18:35:24.909447 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.965468) v2 -- ?+0 0x60f68c0 con 0x618fe40 -9672> 2014-05-19 18:35:24.909486 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9671> 2014-05-19 18:35:24.909520 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.965468) v2 -- ?+0 0x64ae1c0 con 0x61ae420 -9670> 2014-05-19 18:35:24.909559 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9669> 2014-05-19 18:35:24.943696 7fd0396a8700 20 OSD::command_tp worker waiting -9668> 2014-05-19 18:35:25.057820 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 47 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:25.057609) v2 ==== 47+0+0 (1342762915 0 0) 0x55f48c0 con 0x4cb82c0 -9667> 2014-05-19 18:35:25.057832 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 47 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:25.057609) v2 ==== 47+0+0 (1342762915 0 0) 0x70a9340 con 0x4cb8160 -9666> 2014-05-19 18:35:25.057873 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.057609) v2 -- ?+0 0x75b0e00 con 0x4cb82c0 -9665> 2014-05-19 18:35:25.057912 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9664> 2014-05-19 18:35:25.057944 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.057609) v2 -- ?+0 0x5fc3880 con 0x4cb8160 -9663> 2014-05-19 18:35:25.058044 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9662> 2014-05-19 18:35:25.338013 7fd041eb9700 20 OSD::op_tp worker waiting -9661> 2014-05-19 18:35:25.338038 7fd03e6b2700 20 OSD::op_tp worker waiting -9660> 2014-05-19 18:35:25.338057 7fd03c6ae700 20 OSD::op_tp worker waiting -9659> 2014-05-19 18:35:25.338745 7fd03d6b0700 20 OSD::op_tp worker waiting -9658> 2014-05-19 18:35:25.338839 7fd0436bc700 20 OSD::op_tp worker waiting -9657> 2014-05-19 18:35:25.338866 7fd03bead700 20 OSD::op_tp worker waiting -9656> 2014-05-19 18:35:25.338888 7fd03f6b4700 20 OSD::op_tp worker waiting -9655> 2014-05-19 18:35:25.352420 7fd0446be700 20 OSD::op_tp worker waiting -9654> 2014-05-19 18:35:25.352438 7fd03feb5700 20 OSD::op_tp worker waiting -9653> 2014-05-19 18:35:25.359700 7fd040eb7700 20 OSD::op_tp worker waiting -9652> 2014-05-19 18:35:25.364716 7fd0406b6700 20 OSD::op_tp worker waiting -9651> 2014-05-19 18:35:25.386358 7fd039ea9700 20 OSD::disk_tp worker waiting -9650> 2014-05-19 18:35:25.389779 7fd03aeab700 20 OSD::op_tp worker waiting -9649> 2014-05-19 18:35:25.389804 7fd03eeb3700 20 OSD::op_tp worker waiting -9648> 2014-05-19 18:35:25.407809 7fd042ebb700 20 OSD::op_tp worker waiting -9647> 2014-05-19 18:35:25.407833 7fd03deb1700 20 OSD::op_tp worker waiting -9646> 2014-05-19 18:35:25.416176 7fd04e104700 20 KeyValueStore::op_tp worker waiting -9645> 2014-05-19 18:35:25.416179 7fd0426ba700 20 OSD::op_tp worker waiting -9644> 2014-05-19 18:35:25.416215 7fd03ceaf700 20 OSD::op_tp worker waiting -9643> 2014-05-19 18:35:25.416236 7fd0416b8700 20 OSD::op_tp worker waiting -9642> 2014-05-19 18:35:25.416287 7fd03b6ac700 20 OSD::op_tp worker waiting -9641> 2014-05-19 18:35:25.442129 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9640> 2014-05-19 18:35:25.442145 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9639> 2014-05-19 18:35:25.442153 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x5f0ba40 con 0x3622580 -9638> 2014-05-19 18:35:25.442181 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x7999880 con 0x3622840 -9637> 2014-05-19 18:35:25.442214 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x692d500 con 0x3622b00 -9636> 2014-05-19 18:35:25.442228 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x63781c0 con 0x3622dc0 -9635> 2014-05-19 18:35:25.442242 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x3511a40 con 0x30594a0 -9634> 2014-05-19 18:35:25.442269 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x6b70540 con 0x3622f20 -9633> 2014-05-19 18:35:25.442342 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x6543180 con 0x5e7ef20 -9632> 2014-05-19 18:35:25.442374 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x7733a40 con 0x5e7f1e0 -9631> 2014-05-19 18:35:25.442439 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x6fe61c0 con 0x5e7e2c0 -9630> 2014-05-19 18:35:25.442463 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x7746c40 con 0x5d7bb80 -9629> 2014-05-19 18:35:25.442504 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x5b28540 con 0x5e7ec60 -9628> 2014-05-19 18:35:25.442531 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x5496c40 con 0x531b760 -9627> 2014-05-19 18:35:25.442546 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x76ac1c0 con 0x53bba20 -9626> 2014-05-19 18:35:25.442570 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x5f7ac40 con 0x51cac60 -9625> 2014-05-19 18:35:25.442557 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x7998fc0 con 0x3622b00 -9624> 2014-05-19 18:35:25.442616 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x5f07340 con 0x5bd66e0 -9623> 2014-05-19 18:35:25.442632 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.442152) v2 -- ?+0 0x55f4fc0 con 0x5bd6840 -9622> 2014-05-19 18:35:25.442676 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9621> 2014-05-19 18:35:25.442733 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x6e19c00 con 0x3622580 -9620> 2014-05-19 18:35:25.442774 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9619> 2014-05-19 18:35:25.442796 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x6efe540 con 0x3622840 -9618> 2014-05-19 18:35:25.442829 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9617> 2014-05-19 18:35:25.442846 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x7178c40 con 0x3622dc0 -9616> 2014-05-19 18:35:25.442873 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9615> 2014-05-19 18:35:25.442904 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x7733c00 con 0x5e7ef20 -9614> 2014-05-19 18:35:25.442933 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9613> 2014-05-19 18:35:25.442947 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x6c4d180 con 0x5e7f1e0 -9612> 2014-05-19 18:35:25.442999 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9611> 2014-05-19 18:35:25.443017 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x7a5f340 con 0x5e7e2c0 -9610> 2014-05-19 18:35:25.443049 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9609> 2014-05-19 18:35:25.443063 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x6009180 con 0x5d7bb80 -9608> 2014-05-19 18:35:25.443095 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9607> 2014-05-19 18:35:25.443109 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x788a700 con 0x5e7ec60 -9606> 2014-05-19 18:35:25.443136 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9605> 2014-05-19 18:35:25.443153 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x66f9dc0 con 0x531b760 -9604> 2014-05-19 18:35:25.443177 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9603> 2014-05-19 18:35:25.443187 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x6b71500 con 0x53bba20 -9602> 2014-05-19 18:35:25.443205 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9601> 2014-05-19 18:35:25.443210 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x60f61c0 con 0x51cac60 -9600> 2014-05-19 18:35:25.443225 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9599> 2014-05-19 18:35:25.443231 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x5e28c40 con 0x5bd66e0 -9598> 2014-05-19 18:35:25.443248 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9597> 2014-05-19 18:35:25.443256 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 46 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x72e5c00 con 0x5bd6840 -9596> 2014-05-19 18:35:25.443271 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9595> 2014-05-19 18:35:25.443278 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x7971500 con 0x30594a0 -9594> 2014-05-19 18:35:25.443293 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9593> 2014-05-19 18:35:25.443301 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.442152) v2 ==== 47+0+0 (2364959723 0 0) 0x51a7dc0 con 0x3622f20 -9592> 2014-05-19 18:35:25.443315 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9591> 2014-05-19 18:35:25.481719 7fd04f907700 5 osd.0 21 tick -9590> 2014-05-19 18:35:25.481741 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -9589> 2014-05-19 18:35:25.481746 7fd04f907700 10 osd.0 21 do_waiters -- start -9588> 2014-05-19 18:35:25.481748 7fd04f907700 10 osd.0 21 do_waiters -- finish -9587> 2014-05-19 18:35:25.481777 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9586> 2014-05-19 18:35:25.562999 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:30.646633) v2 ==== 47+0+0 (1611258350 0 0) 0x5519500 con 0x31e1600 -9585> 2014-05-19 18:35:25.563022 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:30.646633) v2 ==== 47+0+0 (1611258350 0 0) 0x5c04000 con 0x61b0160 -9584> 2014-05-19 18:35:25.563044 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:30.646633) v2 -- ?+0 0x70a9340 con 0x31e1600 -9583> 2014-05-19 18:35:25.563079 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9582> 2014-05-19 18:35:25.563103 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:30.646633) v2 -- ?+0 0x55f48c0 con 0x61b0160 -9581> 2014-05-19 18:35:25.563153 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9580> 2014-05-19 18:35:25.942772 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9579> 2014-05-19 18:35:25.942790 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9578> 2014-05-19 18:35:25.942798 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x5519880 con 0x3622580 -9577> 2014-05-19 18:35:25.942823 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x6543a40 con 0x3622840 -9576> 2014-05-19 18:35:25.942860 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x5c46700 con 0x3622b00 -9575> 2014-05-19 18:35:25.942874 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x60081c0 con 0x3622dc0 -9574> 2014-05-19 18:35:25.942887 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x6504e00 con 0x30594a0 -9573> 2014-05-19 18:35:25.942913 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x5f328c0 con 0x3622f20 -9572> 2014-05-19 18:35:25.942952 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x59bd500 con 0x5e7ef20 -9571> 2014-05-19 18:35:25.942978 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x3438380 con 0x5e7f1e0 -9570> 2014-05-19 18:35:25.943047 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x64ae000 con 0x5e7e2c0 -9569> 2014-05-19 18:35:25.943077 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x612c8c0 con 0x5d7bb80 -9568> 2014-05-19 18:35:25.943117 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x5497a40 con 0x5e7ec60 -9567> 2014-05-19 18:35:25.943144 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x76ac000 con 0x531b760 -9566> 2014-05-19 18:35:25.943183 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x76adc00 con 0x3622580 -9565> 2014-05-19 18:35:25.943207 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x7998700 con 0x53bba20 -9564> 2014-05-19 18:35:25.943230 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x5f06540 con 0x51cac60 -9563> 2014-05-19 18:35:25.943246 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x5b28380 con 0x5bd66e0 -9562> 2014-05-19 18:35:25.943259 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:25.942797) v2 -- ?+0 0x5c47880 con 0x5bd6840 -9561> 2014-05-19 18:35:25.943318 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9560> 2014-05-19 18:35:25.943345 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x72dfdc0 con 0x3622dc0 -9559> 2014-05-19 18:35:25.943369 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9558> 2014-05-19 18:35:25.943380 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x51a6a80 con 0x3622f20 -9557> 2014-05-19 18:35:25.943397 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9556> 2014-05-19 18:35:25.943410 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x653ea80 con 0x3622840 -9555> 2014-05-19 18:35:25.943458 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9554> 2014-05-19 18:35:25.943477 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x7998a80 con 0x3622b00 -9553> 2014-05-19 18:35:25.943497 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9552> 2014-05-19 18:35:25.943504 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x7732540 con 0x5e7ef20 -9551> 2014-05-19 18:35:25.943519 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9550> 2014-05-19 18:35:25.943524 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x6c4ca80 con 0x5e7f1e0 -9549> 2014-05-19 18:35:25.943541 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9548> 2014-05-19 18:35:25.943551 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x64d2a80 con 0x5e7e2c0 -9547> 2014-05-19 18:35:25.943576 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9546> 2014-05-19 18:35:25.943583 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x6008e00 con 0x5d7bb80 -9545> 2014-05-19 18:35:25.943600 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9544> 2014-05-19 18:35:25.943610 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x788b180 con 0x5e7ec60 -9543> 2014-05-19 18:35:25.943637 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9542> 2014-05-19 18:35:25.943654 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x66f8540 con 0x531b760 -9541> 2014-05-19 18:35:25.943686 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9540> 2014-05-19 18:35:25.943698 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x772d340 con 0x53bba20 -9539> 2014-05-19 18:35:25.943716 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9538> 2014-05-19 18:35:25.943727 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x5c05a40 con 0x51cac60 -9537> 2014-05-19 18:35:25.943742 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9536> 2014-05-19 18:35:25.943750 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x5e28380 con 0x5bd66e0 -9535> 2014-05-19 18:35:25.943765 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9534> 2014-05-19 18:35:25.943773 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 47 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x72e4c40 con 0x5bd6840 -9533> 2014-05-19 18:35:25.943787 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9532> 2014-05-19 18:35:25.943795 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:25.942797) v2 ==== 47+0+0 (2571130367 0 0) 0x7970540 con 0x30594a0 -9531> 2014-05-19 18:35:25.943810 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9530> 2014-05-19 18:35:26.273299 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 44 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:26.273133) v2 ==== 47+0+0 (4281107704 0 0) 0x77e8380 con 0x4cb86e0 -9529> 2014-05-19 18:35:26.273348 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:26.273133) v2 -- ?+0 0x5519500 con 0x4cb86e0 -9528> 2014-05-19 18:35:26.273373 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9527> 2014-05-19 18:35:26.273345 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 44 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:26.273133) v2 ==== 47+0+0 (4281107704 0 0) 0x649ea80 con 0x4cb8420 -9526> 2014-05-19 18:35:26.273388 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:26.273133) v2 -- ?+0 0x5c04000 con 0x4cb8420 -9525> 2014-05-19 18:35:26.273491 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9524> 2014-05-19 18:35:26.481831 7fd04f907700 5 osd.0 21 tick -9523> 2014-05-19 18:35:26.481852 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -9522> 2014-05-19 18:35:26.481856 7fd04f907700 10 osd.0 21 do_waiters -- start -9521> 2014-05-19 18:35:26.481859 7fd04f907700 10 osd.0 21 do_waiters -- finish -9520> 2014-05-19 18:35:26.481898 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9519> 2014-05-19 18:35:26.771461 7fd043ebd700 20 OSD::op_tp worker waiting -9518> 2014-05-19 18:35:26.771696 7fd04e905700 20 KeyValueStore::op_tp worker waiting -9517> 2014-05-19 18:35:26.780001 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 48 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:31.863655) v2 ==== 47+0+0 (430933723 0 0) 0x5b281c0 con 0x5b21600 -9516> 2014-05-19 18:35:26.780045 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.863655) v2 -- ?+0 0x77e8380 con 0x5b21600 -9515> 2014-05-19 18:35:26.780081 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9514> 2014-05-19 18:35:26.780113 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 48 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:31.863655) v2 ==== 47+0+0 (430933723 0 0) 0x5c05880 con 0x5f5b760 -9513> 2014-05-19 18:35:26.780148 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.863655) v2 -- ?+0 0x649ea80 con 0x5f5b760 -9512> 2014-05-19 18:35:26.780188 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9511> 2014-05-19 18:35:26.943791 7fd0396a8700 20 OSD::command_tp worker waiting -9510> 2014-05-19 18:35:27.209869 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:30.265966) v2 ==== 47+0+0 (1250145207 0 0) 0x7947dc0 con 0x61ae420 -9509> 2014-05-19 18:35:27.209886 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:30.265966) v2 ==== 47+0+0 (1250145207 0 0) 0x75b1c00 con 0x618fe40 -9508> 2014-05-19 18:35:27.209924 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:30.265966) v2 -- ?+0 0x5b281c0 con 0x61ae420 -9507> 2014-05-19 18:35:27.209957 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9506> 2014-05-19 18:35:27.209992 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:30.265966) v2 -- ?+0 0x5c05880 con 0x618fe40 -9505> 2014-05-19 18:35:27.210077 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9504> 2014-05-19 18:35:27.338111 7fd041eb9700 20 OSD::op_tp worker waiting -9503> 2014-05-19 18:35:27.338131 7fd03c6ae700 20 OSD::op_tp worker waiting -9502> 2014-05-19 18:35:27.338150 7fd03e6b2700 20 OSD::op_tp worker waiting -9501> 2014-05-19 18:35:27.338834 7fd03d6b0700 20 OSD::op_tp worker waiting -9500> 2014-05-19 18:35:27.338921 7fd0436bc700 20 OSD::op_tp worker waiting -9499> 2014-05-19 18:35:27.338947 7fd03bead700 20 OSD::op_tp worker waiting -9498> 2014-05-19 18:35:27.338969 7fd03f6b4700 20 OSD::op_tp worker waiting -9497> 2014-05-19 18:35:27.352501 7fd0446be700 20 OSD::op_tp worker waiting -9496> 2014-05-19 18:35:27.352520 7fd03feb5700 20 OSD::op_tp worker waiting -9495> 2014-05-19 18:35:27.358359 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 48 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:27.358204) v2 ==== 47+0+0 (1371410939 0 0) 0x55f56c0 con 0x4cb82c0 -9494> 2014-05-19 18:35:27.358408 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.358204) v2 -- ?+0 0x75b1c00 con 0x4cb82c0 -9493> 2014-05-19 18:35:27.358458 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9492> 2014-05-19 18:35:27.358438 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 48 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:27.358204) v2 ==== 47+0+0 (1371410939 0 0) 0x70a9a40 con 0x4cb8160 -9491> 2014-05-19 18:35:27.358483 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.358204) v2 -- ?+0 0x7947dc0 con 0x4cb8160 -9490> 2014-05-19 18:35:27.358516 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9489> 2014-05-19 18:35:27.359769 7fd040eb7700 20 OSD::op_tp worker waiting -9488> 2014-05-19 18:35:27.364798 7fd0406b6700 20 OSD::op_tp worker waiting -9487> 2014-05-19 18:35:27.386454 7fd039ea9700 20 OSD::disk_tp worker waiting -9486> 2014-05-19 18:35:27.389879 7fd03aeab700 20 OSD::op_tp worker waiting -9485> 2014-05-19 18:35:27.389904 7fd03eeb3700 20 OSD::op_tp worker waiting -9484> 2014-05-19 18:35:27.407881 7fd042ebb700 20 OSD::op_tp worker waiting -9483> 2014-05-19 18:35:27.407910 7fd03deb1700 20 OSD::op_tp worker waiting -9482> 2014-05-19 18:35:27.416283 7fd04e104700 20 KeyValueStore::op_tp worker waiting -9481> 2014-05-19 18:35:27.416286 7fd0426ba700 20 OSD::op_tp worker waiting -9480> 2014-05-19 18:35:27.416322 7fd03ceaf700 20 OSD::op_tp worker waiting -9479> 2014-05-19 18:35:27.416342 7fd0416b8700 20 OSD::op_tp worker waiting -9478> 2014-05-19 18:35:27.416369 7fd03b6ac700 20 OSD::op_tp worker waiting -9477> 2014-05-19 18:35:27.481944 7fd04f907700 5 osd.0 21 tick -9476> 2014-05-19 18:35:27.481961 7fd04f907700 7 osd.0 21 do_mon_report -9475> 2014-05-19 18:35:27.481965 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -9474> 2014-05-19 18:35:27.481969 7fd04f907700 20 osd.0 21 send_pg_stats -9473> 2014-05-19 18:35:27.481972 7fd04f907700 10 osd.0 21 send_pg_stats - 0 pgs updated -9472> 2014-05-19 18:35:27.481976 7fd04f907700 10 monclient: _send_mon_message to mon.a at 172.16.96.201:6789/0 -9471> 2014-05-19 18:35:27.481985 7fd04f907700 1 -- 172.16.96.201:6800/3074 --> 172.16.96.201:6789/0 -- pg_stats(0 pgs tid 7 v 0) v1 -- ?+0 0x6b9bb80 con 0x2e3edc0 -9470> 2014-05-19 18:35:27.482004 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -9469> 2014-05-19 18:35:27.482008 7fd04f907700 10 osd.0 21 do_waiters -- start -9468> 2014-05-19 18:35:27.482010 7fd04f907700 10 osd.0 21 do_waiters -- finish -9467> 2014-05-19 18:35:27.482044 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9466> 2014-05-19 18:35:27.643391 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9465> 2014-05-19 18:35:27.643409 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9464> 2014-05-19 18:35:27.643417 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x6542000 con 0x3622580 -9463> 2014-05-19 18:35:27.643443 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x612dc00 con 0x3622840 -9462> 2014-05-19 18:35:27.643480 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x59bcfc0 con 0x3622b00 -9461> 2014-05-19 18:35:27.643494 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x6009c00 con 0x3622dc0 -9460> 2014-05-19 18:35:27.643508 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x6c4d180 con 0x30594a0 -9459> 2014-05-19 18:35:27.643535 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x7733c00 con 0x3622f20 -9458> 2014-05-19 18:35:27.643564 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x7178c40 con 0x5e7ef20 -9457> 2014-05-19 18:35:27.643594 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x6efe540 con 0x5e7f1e0 -9456> 2014-05-19 18:35:27.643666 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x6e19c00 con 0x5e7e2c0 -9455> 2014-05-19 18:35:27.643689 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x7998fc0 con 0x5d7bb80 -9454> 2014-05-19 18:35:27.643723 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x6c4da40 con 0x5e7ec60 -9453> 2014-05-19 18:35:27.643751 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x5497340 con 0x531b760 -9452> 2014-05-19 18:35:27.643785 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x76addc0 con 0x53bba20 -9451> 2014-05-19 18:35:27.643809 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x60f6e00 con 0x51cac60 -9450> 2014-05-19 18:35:27.643836 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x70a81c0 con 0x5bd66e0 -9449> 2014-05-19 18:35:27.643863 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:27.643415) v2 -- ?+0 0x66f9a40 con 0x5bd6840 -9448> 2014-05-19 18:35:27.643890 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x5496000 con 0x3622580 -9447> 2014-05-19 18:35:27.643929 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9446> 2014-05-19 18:35:27.643944 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x7999dc0 con 0x3622b00 -9445> 2014-05-19 18:35:27.643973 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9444> 2014-05-19 18:35:27.644012 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x72de380 con 0x3622dc0 -9443> 2014-05-19 18:35:27.644086 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9442> 2014-05-19 18:35:27.644099 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x77321c0 con 0x5e7ef20 -9441> 2014-05-19 18:35:27.644117 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9440> 2014-05-19 18:35:27.644123 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x6c4c540 con 0x5e7f1e0 -9439> 2014-05-19 18:35:27.644140 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9438> 2014-05-19 18:35:27.644145 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x64d3180 con 0x5e7e2c0 -9437> 2014-05-19 18:35:27.644160 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9436> 2014-05-19 18:35:27.644172 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x6008000 con 0x5d7bb80 -9435> 2014-05-19 18:35:27.644190 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9434> 2014-05-19 18:35:27.644198 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x7970e00 con 0x30594a0 -9433> 2014-05-19 18:35:27.644228 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9432> 2014-05-19 18:35:27.644243 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x51a68c0 con 0x3622f20 -9431> 2014-05-19 18:35:27.644275 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9430> 2014-05-19 18:35:27.644303 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x653e8c0 con 0x3622840 -9429> 2014-05-19 18:35:27.644332 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9428> 2014-05-19 18:35:27.644345 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x788a000 con 0x5e7ec60 -9427> 2014-05-19 18:35:27.644362 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9426> 2014-05-19 18:35:27.644370 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x66f9c00 con 0x531b760 -9425> 2014-05-19 18:35:27.644386 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9424> 2014-05-19 18:35:27.644395 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x77e81c0 con 0x53bba20 -9423> 2014-05-19 18:35:27.644410 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9422> 2014-05-19 18:35:27.644415 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x691b6c0 con 0x51cac60 -9421> 2014-05-19 18:35:27.644429 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9420> 2014-05-19 18:35:27.644437 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x5e288c0 con 0x5bd66e0 -9419> 2014-05-19 18:35:27.644451 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9418> 2014-05-19 18:35:27.644459 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 48 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.643415) v2 ==== 47+0+0 (1458109973 0 0) 0x72e4fc0 con 0x5bd6840 -9417> 2014-05-19 18:35:27.644477 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9416> 2014-05-19 18:35:27.705052 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 45 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:32.788818) v2 ==== 47+0+0 (876917450 0 0) 0x5f0b880 con 0x30946e0 -9415> 2014-05-19 18:35:27.705095 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:32.788818) v2 -- ?+0 0x70a9a40 con 0x30946e0 -9414> 2014-05-19 18:35:27.705172 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9413> 2014-05-19 18:35:27.705152 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 45 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:32.788818) v2 ==== 47+0+0 (876917450 0 0) 0x5dff340 con 0x3094160 -9412> 2014-05-19 18:35:27.705219 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:32.788818) v2 -- ?+0 0x55f56c0 con 0x3094160 -9411> 2014-05-19 18:35:27.705254 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9410> 2014-05-19 18:35:27.710323 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:30.766459) v2 ==== 47+0+0 (1138353888 0 0) 0x649e000 con 0x61ae420 -9409> 2014-05-19 18:35:27.710369 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:30.766459) v2 ==== 47+0+0 (1138353888 0 0) 0x6b71dc0 con 0x618fe40 -9408> 2014-05-19 18:35:27.710373 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:30.766459) v2 -- ?+0 0x5f0b880 con 0x61ae420 -9407> 2014-05-19 18:35:27.710423 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9406> 2014-05-19 18:35:27.710446 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:30.766459) v2 -- ?+0 0x5dff340 con 0x618fe40 -9405> 2014-05-19 18:35:27.710485 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9404> 2014-05-19 18:35:27.858904 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:27.858727) v2 ==== 47+0+0 (3828823974 0 0) 0x6711500 con 0x4cb82c0 -9403> 2014-05-19 18:35:27.858924 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:27.858727) v2 ==== 47+0+0 (3828823974 0 0) 0x70a88c0 con 0x4cb8160 -9402> 2014-05-19 18:35:27.858949 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.858727) v2 -- ?+0 0x6b71dc0 con 0x4cb82c0 -9401> 2014-05-19 18:35:27.858981 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9400> 2014-05-19 18:35:27.859003 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:27.858727) v2 -- ?+0 0x649e000 con 0x4cb8160 -9399> 2014-05-19 18:35:27.859043 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9398> 2014-05-19 18:35:28.047969 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:31.104081) v2 ==== 47+0+0 (819756548 0 0) 0x79e76c0 con 0x4cb8dc0 -9397> 2014-05-19 18:35:28.047972 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:31.104081) v2 ==== 47+0+0 (819756548 0 0) 0x784ca80 con 0x4cb9340 -9396> 2014-05-19 18:35:28.048032 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.104081) v2 -- ?+0 0x70a88c0 con 0x4cb8dc0 -9395> 2014-05-19 18:35:28.048062 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9394> 2014-05-19 18:35:28.048088 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.104081) v2 -- ?+0 0x6711500 con 0x4cb9340 -9393> 2014-05-19 18:35:28.048116 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9392> 2014-05-19 18:35:28.144007 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9391> 2014-05-19 18:35:28.144026 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9390> 2014-05-19 18:35:28.144033 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x692da40 con 0x3622580 -9389> 2014-05-19 18:35:28.144058 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x6009dc0 con 0x3622840 -9388> 2014-05-19 18:35:28.144096 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x3511500 con 0x3622b00 -9387> 2014-05-19 18:35:28.144110 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x7a5e1c0 con 0x3622dc0 -9386> 2014-05-19 18:35:28.144124 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x7733dc0 con 0x30594a0 -9385> 2014-05-19 18:35:28.144150 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x6efefc0 con 0x3622f20 -9384> 2014-05-19 18:35:28.144179 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x6b716c0 con 0x5e7ef20 -9383> 2014-05-19 18:35:28.144240 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x6fe7a40 con 0x5e7f1e0 -9382> 2014-05-19 18:35:28.144312 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x7999180 con 0x5e7e2c0 -9381> 2014-05-19 18:35:28.144354 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x77476c0 con 0x5d7bb80 -9380> 2014-05-19 18:35:28.144420 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x5497dc0 con 0x5e7ec60 -9379> 2014-05-19 18:35:28.144405 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x511ddc0 con 0x3622580 -9378> 2014-05-19 18:35:28.144446 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x76acfc0 con 0x531b760 -9377> 2014-05-19 18:35:28.144461 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x5f7a8c0 con 0x53bba20 -9376> 2014-05-19 18:35:28.144475 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x70a8540 con 0x51cac60 -9375> 2014-05-19 18:35:28.144493 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x66f81c0 con 0x5bd66e0 -9374> 2014-05-19 18:35:28.144513 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:28.144032) v2 -- ?+0 0x692dc00 con 0x5bd6840 -9373> 2014-05-19 18:35:28.144569 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9372> 2014-05-19 18:35:28.144616 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x7998c40 con 0x3622b00 -9371> 2014-05-19 18:35:28.144657 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9370> 2014-05-19 18:35:28.144669 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x5f7b180 con 0x3622dc0 -9369> 2014-05-19 18:35:28.144697 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9368> 2014-05-19 18:35:28.144716 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x6b71c00 con 0x5e7ef20 -9367> 2014-05-19 18:35:28.144749 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9366> 2014-05-19 18:35:28.144764 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x6c4c700 con 0x5e7f1e0 -9365> 2014-05-19 18:35:28.144795 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9364> 2014-05-19 18:35:28.144804 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x60088c0 con 0x5d7bb80 -9363> 2014-05-19 18:35:28.144834 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9362> 2014-05-19 18:35:28.144850 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x79716c0 con 0x30594a0 -9361> 2014-05-19 18:35:28.144879 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9360> 2014-05-19 18:35:28.144916 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x6543500 con 0x3622f20 -9359> 2014-05-19 18:35:28.144962 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9358> 2014-05-19 18:35:28.144973 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x653efc0 con 0x3622840 -9357> 2014-05-19 18:35:28.144990 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9356> 2014-05-19 18:35:28.144996 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x55c28c0 con 0x5e7e2c0 -9355> 2014-05-19 18:35:28.145010 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9354> 2014-05-19 18:35:28.145019 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x788a540 con 0x5e7ec60 -9353> 2014-05-19 18:35:28.145034 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9352> 2014-05-19 18:35:28.145043 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x66f8700 con 0x531b760 -9351> 2014-05-19 18:35:28.145057 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9350> 2014-05-19 18:35:28.145073 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x5fc3180 con 0x53bba20 -9349> 2014-05-19 18:35:28.145090 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9348> 2014-05-19 18:35:28.145095 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x5667a40 con 0x51cac60 -9347> 2014-05-19 18:35:28.145127 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9346> 2014-05-19 18:35:28.145136 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x5e28e00 con 0x5bd66e0 -9345> 2014-05-19 18:35:28.145151 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9344> 2014-05-19 18:35:28.145157 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 49 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.144032) v2 ==== 47+0+0 (4142430099 0 0) 0x73c2000 con 0x5bd6840 -9343> 2014-05-19 18:35:28.145171 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9342> 2014-05-19 18:35:28.364909 7fd04aecb700 1 -- 172.16.96.201:6800/3074 <== mon.0 172.16.96.201:6789/0 43 ==== pg_stats_ack(0 pgs tid 7) v1 ==== 4+0+0 (0 0 0) 0x8389180 con 0x2e3edc0 -9341> 2014-05-19 18:35:28.364940 7fd04aecb700 10 osd.0 21 do_waiters -- start -9340> 2014-05-19 18:35:28.364945 7fd04aecb700 10 osd.0 21 do_waiters -- finish -9339> 2014-05-19 18:35:28.364947 7fd04aecb700 20 osd.0 21 _dispatch 0x8389180 pg_stats_ack(0 pgs tid 7) v1 -9338> 2014-05-19 18:35:28.364953 7fd04aecb700 10 osd.0 21 handle_pg_stats_ack -9337> 2014-05-19 18:35:28.364957 7fd04aecb700 10 osd.0 21 do_waiters -- start -9336> 2014-05-19 18:35:28.364959 7fd04aecb700 10 osd.0 21 do_waiters -- finish -9335> 2014-05-19 18:35:28.482091 7fd04f907700 5 osd.0 21 tick -9334> 2014-05-19 18:35:28.482112 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -9333> 2014-05-19 18:35:28.482117 7fd04f907700 10 osd.0 21 do_waiters -- start -9332> 2014-05-19 18:35:28.482119 7fd04f907700 10 osd.0 21 do_waiters -- finish -9331> 2014-05-19 18:35:28.482160 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9330> 2014-05-19 18:35:28.573862 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 45 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:28.573674) v2 ==== 47+0+0 (3056326007 0 0) 0x55c8700 con 0x4cb86e0 -9329> 2014-05-19 18:35:28.573903 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.573674) v2 -- ?+0 0x79e76c0 con 0x4cb86e0 -9328> 2014-05-19 18:35:28.573934 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9327> 2014-05-19 18:35:28.573952 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 45 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:28.573674) v2 ==== 47+0+0 (3056326007 0 0) 0x649e540 con 0x4cb8420 -9326> 2014-05-19 18:35:28.573983 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:28.573674) v2 -- ?+0 0x784ca80 con 0x4cb8420 -9325> 2014-05-19 18:35:28.574012 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9324> 2014-05-19 18:35:28.771545 7fd043ebd700 20 OSD::op_tp worker waiting -9323> 2014-05-19 18:35:28.771763 7fd04e905700 20 KeyValueStore::op_tp worker waiting -9322> 2014-05-19 18:35:28.943883 7fd0396a8700 20 OSD::command_tp worker waiting -9321> 2014-05-19 18:35:29.338167 7fd041eb9700 20 OSD::op_tp worker waiting -9320> 2014-05-19 18:35:29.338196 7fd03c6ae700 20 OSD::op_tp worker waiting -9319> 2014-05-19 18:35:29.338226 7fd03e6b2700 20 OSD::op_tp worker waiting -9318> 2014-05-19 18:35:29.338903 7fd03d6b0700 20 OSD::op_tp worker waiting -9317> 2014-05-19 18:35:29.339003 7fd0436bc700 20 OSD::op_tp worker waiting -9316> 2014-05-19 18:35:29.339025 7fd03bead700 20 OSD::op_tp worker waiting -9315> 2014-05-19 18:35:29.339050 7fd03f6b4700 20 OSD::op_tp worker waiting -9314> 2014-05-19 18:35:29.352594 7fd0446be700 20 OSD::op_tp worker waiting -9313> 2014-05-19 18:35:29.352621 7fd03feb5700 20 OSD::op_tp worker waiting -9312> 2014-05-19 18:35:29.359840 7fd040eb7700 20 OSD::op_tp worker waiting -9311> 2014-05-19 18:35:29.364880 7fd0406b6700 20 OSD::op_tp worker waiting -9310> 2014-05-19 18:35:29.386550 7fd039ea9700 20 OSD::disk_tp worker waiting -9309> 2014-05-19 18:35:29.389979 7fd03aeab700 20 OSD::op_tp worker waiting -9308> 2014-05-19 18:35:29.390003 7fd03eeb3700 20 OSD::op_tp worker waiting -9307> 2014-05-19 18:35:29.407978 7fd042ebb700 20 OSD::op_tp worker waiting -9306> 2014-05-19 18:35:29.408002 7fd03deb1700 20 OSD::op_tp worker waiting -9305> 2014-05-19 18:35:29.410872 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:32.466983) v2 ==== 47+0+0 (3228445581 0 0) 0x6b70380 con 0x618fe40 -9304> 2014-05-19 18:35:29.410906 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:32.466983) v2 ==== 47+0+0 (3228445581 0 0) 0x79e7340 con 0x61ae420 -9303> 2014-05-19 18:35:29.410914 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:32.466983) v2 -- ?+0 0x649e540 con 0x618fe40 -9302> 2014-05-19 18:35:29.410973 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9301> 2014-05-19 18:35:29.410999 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:32.466983) v2 -- ?+0 0x55c8700 con 0x61ae420 -9300> 2014-05-19 18:35:29.411030 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9299> 2014-05-19 18:35:29.416379 7fd04e104700 20 KeyValueStore::op_tp worker waiting -9298> 2014-05-19 18:35:29.416381 7fd0426ba700 20 OSD::op_tp worker waiting -9297> 2014-05-19 18:35:29.416417 7fd03ceaf700 20 OSD::op_tp worker waiting -9296> 2014-05-19 18:35:29.416438 7fd0416b8700 20 OSD::op_tp worker waiting -9295> 2014-05-19 18:35:29.416461 7fd03b6ac700 20 OSD::op_tp worker waiting -9294> 2014-05-19 18:35:29.482205 7fd04f907700 5 osd.0 21 tick -9293> 2014-05-19 18:35:29.482225 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -9292> 2014-05-19 18:35:29.482230 7fd04f907700 10 osd.0 21 do_waiters -- start -9291> 2014-05-19 18:35:29.482232 7fd04f907700 10 osd.0 21 do_waiters -- finish -9290> 2014-05-19 18:35:29.482274 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9289> 2014-05-19 18:35:29.559481 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:29.559285) v2 ==== 47+0+0 (1465056651 0 0) 0x67101c0 con 0x4cb82c0 -9288> 2014-05-19 18:35:29.559502 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:29.559285) v2 ==== 47+0+0 (1465056651 0 0) 0x5dea8c0 con 0x4cb8160 -9287> 2014-05-19 18:35:29.559529 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:29.559285) v2 -- ?+0 0x6b70380 con 0x4cb82c0 -9286> 2014-05-19 18:35:29.559566 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9285> 2014-05-19 18:35:29.559623 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:29.559285) v2 -- ?+0 0x79e7340 con 0x4cb8160 -9284> 2014-05-19 18:35:29.559666 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9283> 2014-05-19 18:35:29.906373 7fd0456c0700 10 monclient: tick -9282> 2014-05-19 18:35:29.906385 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:34:59.906384) -9281> 2014-05-19 18:35:29.906402 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:35:29.906402; renew after: 2014-05-19 18:35:39.994233) -- no -9280> 2014-05-19 18:35:29.911360 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:32.967487) v2 ==== 47+0+0 (2990089392 0 0) 0x691b340 con 0x61ae420 -9279> 2014-05-19 18:35:29.911369 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:32.967487) v2 ==== 47+0+0 (2990089392 0 0) 0x70a9dc0 con 0x618fe40 -9278> 2014-05-19 18:35:29.911398 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:32.967487) v2 -- ?+0 0x5dea8c0 con 0x61ae420 -9277> 2014-05-19 18:35:29.911487 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9276> 2014-05-19 18:35:29.911547 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:32.967487) v2 -- ?+0 0x67101c0 con 0x618fe40 -9275> 2014-05-19 18:35:29.911582 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9274> 2014-05-19 18:35:29.978215 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 48 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:33.034351) v2 ==== 47+0+0 (4219461907 0 0) 0x64aec40 con 0x61ae160 -9273> 2014-05-19 18:35:29.978268 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.034351) v2 -- ?+0 0x691b340 con 0x61ae160 -9272> 2014-05-19 18:35:29.978230 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 48 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:33.034351) v2 ==== 47+0+0 (4219461907 0 0) 0x70cd500 con 0x61b0420 -9271> 2014-05-19 18:35:29.978302 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9270> 2014-05-19 18:35:29.978352 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.034351) v2 -- ?+0 0x70a9dc0 con 0x61b0420 -9269> 2014-05-19 18:35:29.978381 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9268> 2014-05-19 18:35:30.060041 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:30.059839) v2 ==== 47+0+0 (263073722 0 0) 0x5debc00 con 0x4cb8160 -9267> 2014-05-19 18:35:30.060052 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:30.059839) v2 ==== 47+0+0 (263073722 0 0) 0x6711180 con 0x4cb82c0 -9266> 2014-05-19 18:35:30.060081 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:30.059839) v2 -- ?+0 0x64aec40 con 0x4cb8160 -9265> 2014-05-19 18:35:30.060156 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9264> 2014-05-19 18:35:30.060183 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:30.059839) v2 -- ?+0 0x70cd500 con 0x4cb82c0 -9263> 2014-05-19 18:35:30.060221 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9262> 2014-05-19 18:35:30.348456 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:33.404597) v2 ==== 47+0+0 (1973919777 0 0) 0x6c4a000 con 0x4cb9340 -9261> 2014-05-19 18:35:30.348470 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:33.404597) v2 ==== 47+0+0 (1973919777 0 0) 0x5dfe8c0 con 0x4cb8dc0 -9260> 2014-05-19 18:35:30.348503 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.404597) v2 -- ?+0 0x6711180 con 0x4cb9340 -9259> 2014-05-19 18:35:30.348538 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9258> 2014-05-19 18:35:30.348598 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.404597) v2 -- ?+0 0x5debc00 con 0x4cb8dc0 -9257> 2014-05-19 18:35:30.348696 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9256> 2014-05-19 18:35:30.482315 7fd04f907700 5 osd.0 21 tick -9255> 2014-05-19 18:35:30.482336 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -9254> 2014-05-19 18:35:30.482340 7fd04f907700 10 osd.0 21 do_waiters -- start -9253> 2014-05-19 18:35:30.482343 7fd04f907700 10 osd.0 21 do_waiters -- finish -9252> 2014-05-19 18:35:30.482366 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9251> 2014-05-19 18:35:30.771626 7fd043ebd700 20 OSD::op_tp worker waiting -9250> 2014-05-19 18:35:30.771829 7fd04e905700 20 KeyValueStore::op_tp worker waiting -9249> 2014-05-19 18:35:30.848953 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:33.905108) v2 ==== 47+0+0 (3956803191 0 0) 0x6c4b180 con 0x4cb9340 -9248> 2014-05-19 18:35:30.848973 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:33.905108) v2 ==== 47+0+0 (3956803191 0 0) 0x5dfee00 con 0x4cb8dc0 -9247> 2014-05-19 18:35:30.849002 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.905108) v2 -- ?+0 0x6c4a000 con 0x4cb9340 -9246> 2014-05-19 18:35:30.849034 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9245> 2014-05-19 18:35:30.849066 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.905108) v2 -- ?+0 0x5dfe8c0 con 0x4cb8dc0 -9244> 2014-05-19 18:35:30.849111 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9243> 2014-05-19 18:35:30.863302 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:35.947122) v2 ==== 47+0+0 (293050575 0 0) 0x5519c00 con 0x31e1600 -9242> 2014-05-19 18:35:30.863304 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:35.947122) v2 ==== 47+0+0 (293050575 0 0) 0x511c380 con 0x61b0160 -9241> 2014-05-19 18:35:30.863357 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:35.947122) v2 -- ?+0 0x5dfee00 con 0x31e1600 -9240> 2014-05-19 18:35:30.863394 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9239> 2014-05-19 18:35:30.863414 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:35.947122) v2 -- ?+0 0x6c4b180 con 0x61b0160 -9238> 2014-05-19 18:35:30.863448 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9237> 2014-05-19 18:35:30.943971 7fd0396a8700 20 OSD::command_tp worker waiting -9236> 2014-05-19 18:35:31.338262 7fd041eb9700 20 OSD::op_tp worker waiting -9235> 2014-05-19 18:35:31.338284 7fd03c6ae700 20 OSD::op_tp worker waiting -9234> 2014-05-19 18:35:31.338299 7fd03e6b2700 20 OSD::op_tp worker waiting -9233> 2014-05-19 18:35:31.338968 7fd03d6b0700 20 OSD::op_tp worker waiting -9232> 2014-05-19 18:35:31.339083 7fd0436bc700 20 OSD::op_tp worker waiting -9231> 2014-05-19 18:35:31.339106 7fd03bead700 20 OSD::op_tp worker waiting -9230> 2014-05-19 18:35:31.339126 7fd03f6b4700 20 OSD::op_tp worker waiting -9229> 2014-05-19 18:35:31.352676 7fd0446be700 20 OSD::op_tp worker waiting -9228> 2014-05-19 18:35:31.352704 7fd03feb5700 20 OSD::op_tp worker waiting -9227> 2014-05-19 18:35:31.359931 7fd040eb7700 20 OSD::op_tp worker waiting -9226> 2014-05-19 18:35:31.364984 7fd0406b6700 20 OSD::op_tp worker waiting -9225> 2014-05-19 18:35:31.386645 7fd039ea9700 20 OSD::disk_tp worker waiting -9224> 2014-05-19 18:35:31.390079 7fd03aeab700 20 OSD::op_tp worker waiting -9223> 2014-05-19 18:35:31.390107 7fd03eeb3700 20 OSD::op_tp worker waiting -9222> 2014-05-19 18:35:31.408078 7fd042ebb700 20 OSD::op_tp worker waiting -9221> 2014-05-19 18:35:31.408100 7fd03deb1700 20 OSD::op_tp worker waiting -9220> 2014-05-19 18:35:31.416473 7fd04e104700 20 KeyValueStore::op_tp worker waiting -9219> 2014-05-19 18:35:31.416474 7fd0426ba700 20 OSD::op_tp worker waiting -9218> 2014-05-19 18:35:31.416510 7fd0416b8700 20 OSD::op_tp worker waiting -9217> 2014-05-19 18:35:31.416526 7fd03ceaf700 20 OSD::op_tp worker waiting -9216> 2014-05-19 18:35:31.416550 7fd03b6ac700 20 OSD::op_tp worker waiting -9215> 2014-05-19 18:35:31.482430 7fd04f907700 5 osd.0 21 tick -9214> 2014-05-19 18:35:31.482482 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.18 < max 0.5 = yes -9213> 2014-05-19 18:35:31.482496 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -9212> 2014-05-19 18:35:31.482499 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9211> 2014-05-19 18:35:31.482499 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 174.207 < min (86400 seconds) -9210> 2014-05-19 18:35:31.482518 7fd04f907700 20 osd.0 21 sched_scrub done -9209> 2014-05-19 18:35:31.482521 7fd04f907700 10 osd.0 21 do_waiters -- start -9208> 2014-05-19 18:35:31.482523 7fd04f907700 10 osd.0 21 do_waiters -- finish -9207> 2014-05-19 18:35:31.644635 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9206> 2014-05-19 18:35:31.644653 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9205> 2014-05-19 18:35:31.644661 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x7a5f180 con 0x3622580 -9204> 2014-05-19 18:35:31.644687 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x6c4d500 con 0x3622840 -9203> 2014-05-19 18:35:31.644724 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x83881c0 con 0x3622b00 -9202> 2014-05-19 18:35:31.644739 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x682f340 con 0x3622dc0 -9201> 2014-05-19 18:35:31.644754 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x64d3180 con 0x30594a0 -9200> 2014-05-19 18:35:31.644780 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x6c4c540 con 0x3622f20 -9199> 2014-05-19 18:35:31.644810 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x77321c0 con 0x5e7ef20 -9198> 2014-05-19 18:35:31.644836 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x72de380 con 0x5e7f1e0 -9197> 2014-05-19 18:35:31.644863 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x7999dc0 con 0x5e7e2c0 -9196> 2014-05-19 18:35:31.644889 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x5496000 con 0x5d7bb80 -9195> 2014-05-19 18:35:31.644915 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x7970540 con 0x5e7ec60 -9194> 2014-05-19 18:35:31.644956 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x72e4c40 con 0x531b760 -9193> 2014-05-19 18:35:31.644977 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x5e28380 con 0x53bba20 -9192> 2014-05-19 18:35:31.645041 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x5c05a40 con 0x51cac60 -9191> 2014-05-19 18:35:31.645082 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x772d340 con 0x5bd66e0 -9190> 2014-05-19 18:35:31.645134 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:31.644659) v2 -- ?+0 0x66f8540 con 0x5bd6840 -9189> 2014-05-19 18:35:31.645064 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x5f7b340 con 0x3622dc0 -9188> 2014-05-19 18:35:31.645186 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9187> 2014-05-19 18:35:31.645217 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x798e700 con 0x3622840 -9186> 2014-05-19 18:35:31.645241 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9185> 2014-05-19 18:35:31.645257 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x682fdc0 con 0x3622b00 -9184> 2014-05-19 18:35:31.645280 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9183> 2014-05-19 18:35:31.645286 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x6e2a380 con 0x5e7f1e0 -9182> 2014-05-19 18:35:31.645301 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9181> 2014-05-19 18:35:31.645311 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x65421c0 con 0x3622f20 -9180> 2014-05-19 18:35:31.645344 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9179> 2014-05-19 18:35:31.645353 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x711e700 con 0x3622580 -9178> 2014-05-19 18:35:31.645370 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9177> 2014-05-19 18:35:31.645382 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x6b71340 con 0x5e7ef20 -9176> 2014-05-19 18:35:31.645413 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9175> 2014-05-19 18:35:31.645421 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x55c3500 con 0x5e7e2c0 -9174> 2014-05-19 18:35:31.645437 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9173> 2014-05-19 18:35:31.645443 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x6008540 con 0x5d7bb80 -9172> 2014-05-19 18:35:31.645460 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9171> 2014-05-19 18:35:31.645468 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x612ca80 con 0x5e7ec60 -9170> 2014-05-19 18:35:31.645483 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9169> 2014-05-19 18:35:31.645513 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x6008380 con 0x53bba20 -9168> 2014-05-19 18:35:31.645546 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9167> 2014-05-19 18:35:31.645559 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x725da40 con 0x30594a0 -9166> 2014-05-19 18:35:31.645577 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9165> 2014-05-19 18:35:31.645585 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x66f8e00 con 0x531b760 -9164> 2014-05-19 18:35:31.645600 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9163> 2014-05-19 18:35:31.645605 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x662bc00 con 0x51cac60 -9162> 2014-05-19 18:35:31.645619 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9161> 2014-05-19 18:35:31.645624 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x72e4700 con 0x5bd66e0 -9160> 2014-05-19 18:35:31.645638 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9159> 2014-05-19 18:35:31.645646 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 50 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:31.644659) v2 ==== 47+0+0 (611484338 0 0) 0x73c28c0 con 0x5bd6840 -9158> 2014-05-19 18:35:31.645662 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9157> 2014-05-19 18:35:32.080252 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:37.164121) v2 ==== 47+0+0 (3121514263 0 0) 0x73c0540 con 0x5b21600 -9156> 2014-05-19 18:35:32.080260 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:37.164121) v2 ==== 47+0+0 (3121514263 0 0) 0x2ee1dc0 con 0x5f5b760 -9155> 2014-05-19 18:35:32.080299 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:37.164121) v2 -- ?+0 0x5519c00 con 0x5b21600 -9154> 2014-05-19 18:35:32.080377 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9153> 2014-05-19 18:35:32.080400 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:37.164121) v2 -- ?+0 0x511c380 con 0x5f5b760 -9152> 2014-05-19 18:35:32.080442 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9151> 2014-05-19 18:35:32.278648 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:35.334813) v2 ==== 47+0+0 (2675631126 0 0) 0x64afdc0 con 0x61ae160 -9150> 2014-05-19 18:35:32.278659 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:35.334813) v2 ==== 47+0+0 (2675631126 0 0) 0x70cda40 con 0x61b0420 -9149> 2014-05-19 18:35:32.278705 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:35.334813) v2 -- ?+0 0x73c0540 con 0x61ae160 -9148> 2014-05-19 18:35:32.278740 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9147> 2014-05-19 18:35:32.278757 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:35.334813) v2 -- ?+0 0x2ee1dc0 con 0x61b0420 -9146> 2014-05-19 18:35:32.278834 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9145> 2014-05-19 18:35:32.482607 7fd04f907700 5 osd.0 21 tick -9144> 2014-05-19 18:35:32.482625 7fd04f907700 7 osd.0 21 do_mon_report -9143> 2014-05-19 18:35:32.482631 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -9142> 2014-05-19 18:35:32.482637 7fd04f907700 20 osd.0 21 send_pg_stats -9141> 2014-05-19 18:35:32.482686 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9140> 2014-05-19 18:35:32.482680 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.18 < max 0.5 = yes -9139> 2014-05-19 18:35:32.482697 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -9138> 2014-05-19 18:35:32.482702 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 175.207 < min (86400 seconds) -9137> 2014-05-19 18:35:32.482737 7fd04f907700 20 osd.0 21 sched_scrub done -9136> 2014-05-19 18:35:32.482742 7fd04f907700 10 osd.0 21 do_waiters -- start -9135> 2014-05-19 18:35:32.482745 7fd04f907700 10 osd.0 21 do_waiters -- finish -9134> 2014-05-19 18:35:32.549448 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:35.605613) v2 ==== 47+0+0 (3059442089 0 0) 0x662b500 con 0x4cb8dc0 -9133> 2014-05-19 18:35:32.549504 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:35.605613) v2 -- ?+0 0x64afdc0 con 0x4cb8dc0 -9132> 2014-05-19 18:35:32.549448 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:35.605613) v2 ==== 47+0+0 (3059442089 0 0) 0x3438a80 con 0x4cb9340 -9131> 2014-05-19 18:35:32.549540 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9130> 2014-05-19 18:35:32.549656 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:35.605613) v2 -- ?+0 0x70cda40 con 0x4cb9340 -9129> 2014-05-19 18:35:32.549708 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9128> 2014-05-19 18:35:32.771722 7fd043ebd700 20 OSD::op_tp worker waiting -9127> 2014-05-19 18:35:32.771899 7fd04e905700 20 KeyValueStore::op_tp worker waiting -9126> 2014-05-19 18:35:32.779246 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:35.835404) v2 ==== 47+0+0 (92263091 0 0) 0x64ae700 con 0x61ae160 -9125> 2014-05-19 18:35:32.779239 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:35.835404) v2 ==== 47+0+0 (92263091 0 0) 0x59bc1c0 con 0x61b0420 -9124> 2014-05-19 18:35:32.779293 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:35.835404) v2 -- ?+0 0x662b500 con 0x61ae160 -9123> 2014-05-19 18:35:32.779365 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9122> 2014-05-19 18:35:32.779440 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:35.835404) v2 -- ?+0 0x3438a80 con 0x61b0420 -9121> 2014-05-19 18:35:32.779501 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9120> 2014-05-19 18:35:32.944056 7fd0396a8700 20 OSD::command_tp worker waiting -9119> 2014-05-19 18:35:33.005334 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 46 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:38.089242) v2 ==== 47+0+0 (1930963021 0 0) 0x662b6c0 con 0x3094160 -9118> 2014-05-19 18:35:33.005382 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:38.089242) v2 -- ?+0 0x59bc1c0 con 0x3094160 -9117> 2014-05-19 18:35:33.005383 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 46 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:38.089242) v2 ==== 47+0+0 (1930963021 0 0) 0x64ae1c0 con 0x30946e0 -9116> 2014-05-19 18:35:33.005415 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9115> 2014-05-19 18:35:33.005455 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:38.089242) v2 -- ?+0 0x64ae700 con 0x30946e0 -9114> 2014-05-19 18:35:33.005489 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9113> 2014-05-19 18:35:33.049960 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:36.106131) v2 ==== 47+0+0 (4003742002 0 0) 0x3438700 con 0x4cb9340 -9112> 2014-05-19 18:35:33.049980 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:36.106131) v2 ==== 47+0+0 (4003742002 0 0) 0x662ba40 con 0x4cb8dc0 -9111> 2014-05-19 18:35:33.050009 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.106131) v2 -- ?+0 0x662b6c0 con 0x4cb9340 -9110> 2014-05-19 18:35:33.050043 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9109> 2014-05-19 18:35:33.050063 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.106131) v2 -- ?+0 0x64ae1c0 con 0x4cb8dc0 -9108> 2014-05-19 18:35:33.050150 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9107> 2014-05-19 18:35:33.163700 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:38.247670) v2 ==== 47+0+0 (2367353316 0 0) 0x772ce00 con 0x31e1600 -9106> 2014-05-19 18:35:33.163751 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:38.247670) v2 -- ?+0 0x662ba40 con 0x31e1600 -9105> 2014-05-19 18:35:33.163804 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:38.247670) v2 ==== 47+0+0 (2367353316 0 0) 0x55c9500 con 0x61b0160 -9104> 2014-05-19 18:35:33.163860 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9103> 2014-05-19 18:35:33.163958 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:38.247670) v2 -- ?+0 0x3438700 con 0x61b0160 -9102> 2014-05-19 18:35:33.163997 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9101> 2014-05-19 18:35:33.338359 7fd041eb9700 20 OSD::op_tp worker waiting -9100> 2014-05-19 18:35:33.338398 7fd03c6ae700 20 OSD::op_tp worker waiting -9099> 2014-05-19 18:35:33.338448 7fd03e6b2700 20 OSD::op_tp worker waiting -9098> 2014-05-19 18:35:33.339036 7fd03d6b0700 20 OSD::op_tp worker waiting -9097> 2014-05-19 18:35:33.339165 7fd0436bc700 20 OSD::op_tp worker waiting -9096> 2014-05-19 18:35:33.339183 7fd03bead700 20 OSD::op_tp worker waiting -9095> 2014-05-19 18:35:33.339203 7fd03f6b4700 20 OSD::op_tp worker waiting -9094> 2014-05-19 18:35:33.345286 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9093> 2014-05-19 18:35:33.345304 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -9092> 2014-05-19 18:35:33.345312 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x788b180 con 0x3622580 -9091> 2014-05-19 18:35:33.345338 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x6008e00 con 0x3622840 -9090> 2014-05-19 18:35:33.345376 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x64d2a80 con 0x3622b00 -9089> 2014-05-19 18:35:33.345391 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x6c4ca80 con 0x3622dc0 -9088> 2014-05-19 18:35:33.345418 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x7732540 con 0x30594a0 -9087> 2014-05-19 18:35:33.345445 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x7998a80 con 0x3622f20 -9086> 2014-05-19 18:35:33.345519 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x653ea80 con 0x5e7ef20 -9085> 2014-05-19 18:35:33.345590 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x51a6a80 con 0x5e7f1e0 -9084> 2014-05-19 18:35:33.345617 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x72dfdc0 con 0x5e7e2c0 -9083> 2014-05-19 18:35:33.345640 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x76adc00 con 0x5d7bb80 -9082> 2014-05-19 18:35:33.345653 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x711ee00 con 0x3622580 -9081> 2014-05-19 18:35:33.345665 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x51a7dc0 con 0x5e7ec60 -9080> 2014-05-19 18:35:33.345724 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x7971500 con 0x531b760 -9079> 2014-05-19 18:35:33.345754 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x72e5c00 con 0x53bba20 -9078> 2014-05-19 18:35:33.345781 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x5e28c40 con 0x51cac60 -9077> 2014-05-19 18:35:33.345808 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x60f61c0 con 0x5bd66e0 -9076> 2014-05-19 18:35:33.345835 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:33.345311) v2 -- ?+0 0x6b71500 con 0x5bd6840 -9075> 2014-05-19 18:35:33.345916 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9074> 2014-05-19 18:35:33.345942 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x6d60700 con 0x3622dc0 -9073> 2014-05-19 18:35:33.345970 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9072> 2014-05-19 18:35:33.345983 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x79988c0 con 0x5e7ef20 -9071> 2014-05-19 18:35:33.345999 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9070> 2014-05-19 18:35:33.346024 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x6e2ba40 con 0x5e7f1e0 -9069> 2014-05-19 18:35:33.346042 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9068> 2014-05-19 18:35:33.346048 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x5518700 con 0x5d7bb80 -9067> 2014-05-19 18:35:33.346063 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9066> 2014-05-19 18:35:33.346071 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x725ca80 con 0x30594a0 -9065> 2014-05-19 18:35:33.346089 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9064> 2014-05-19 18:35:33.346096 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x65436c0 con 0x3622f20 -9063> 2014-05-19 18:35:33.346111 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -9062> 2014-05-19 18:35:33.346116 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x798f880 con 0x3622840 -9061> 2014-05-19 18:35:33.346155 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9060> 2014-05-19 18:35:33.346180 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x682e700 con 0x3622b00 -9059> 2014-05-19 18:35:33.346247 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -9058> 2014-05-19 18:35:33.346256 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x55c3880 con 0x5e7e2c0 -9057> 2014-05-19 18:35:33.346284 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -9056> 2014-05-19 18:35:33.346297 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x612ce00 con 0x5e7ec60 -9055> 2014-05-19 18:35:33.346315 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9054> 2014-05-19 18:35:33.346323 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x66f9500 con 0x531b760 -9053> 2014-05-19 18:35:33.346338 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9052> 2014-05-19 18:35:33.346349 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x2ee01c0 con 0x53bba20 -9051> 2014-05-19 18:35:33.346366 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9050> 2014-05-19 18:35:33.346373 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x79e6c40 con 0x51cac60 -9049> 2014-05-19 18:35:33.346388 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -9048> 2014-05-19 18:35:33.346393 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x72e5500 con 0x5bd66e0 -9047> 2014-05-19 18:35:33.346410 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9046> 2014-05-19 18:35:33.346417 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 51 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.345311) v2 ==== 47+0+0 (1090999088 0 0) 0x73c21c0 con 0x5bd6840 -9045> 2014-05-19 18:35:33.346435 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9044> 2014-05-19 18:35:33.352750 7fd0446be700 20 OSD::op_tp worker waiting -9043> 2014-05-19 18:35:33.352781 7fd03feb5700 20 OSD::op_tp worker waiting -9042> 2014-05-19 18:35:33.360030 7fd040eb7700 20 OSD::op_tp worker waiting -9041> 2014-05-19 18:35:33.365067 7fd0406b6700 20 OSD::op_tp worker waiting -9040> 2014-05-19 18:35:33.386742 7fd039ea9700 20 OSD::disk_tp worker waiting -9039> 2014-05-19 18:35:33.390181 7fd03aeab700 20 OSD::op_tp worker waiting -9038> 2014-05-19 18:35:33.390205 7fd03eeb3700 20 OSD::op_tp worker waiting -9037> 2014-05-19 18:35:33.408177 7fd042ebb700 20 OSD::op_tp worker waiting -9036> 2014-05-19 18:35:33.408204 7fd03deb1700 20 OSD::op_tp worker waiting -9035> 2014-05-19 18:35:33.411861 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:36.468010) v2 ==== 47+0+0 (613127031 0 0) 0x662b180 con 0x61ae420 -9034> 2014-05-19 18:35:33.411868 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:36.468010) v2 ==== 47+0+0 (613127031 0 0) 0x70a8e00 con 0x618fe40 -9033> 2014-05-19 18:35:33.411906 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.468010) v2 -- ?+0 0x772ce00 con 0x61ae420 -9032> 2014-05-19 18:35:33.411942 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9031> 2014-05-19 18:35:33.411961 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.468010) v2 -- ?+0 0x55c9500 con 0x618fe40 -9030> 2014-05-19 18:35:33.411999 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -9029> 2014-05-19 18:35:33.416586 7fd0426ba700 20 OSD::op_tp worker waiting -9028> 2014-05-19 18:35:33.416572 7fd04e104700 20 KeyValueStore::op_tp worker waiting -9027> 2014-05-19 18:35:33.416612 7fd0416b8700 20 OSD::op_tp worker waiting -9026> 2014-05-19 18:35:33.416624 7fd03b6ac700 20 OSD::op_tp worker waiting -9025> 2014-05-19 18:35:33.416650 7fd03ceaf700 20 OSD::op_tp worker waiting -9024> 2014-05-19 18:35:33.482835 7fd04f907700 5 osd.0 21 tick -9023> 2014-05-19 18:35:33.482858 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -9022> 2014-05-19 18:35:33.482865 7fd04f907700 10 osd.0 21 do_waiters -- start -9021> 2014-05-19 18:35:33.482869 7fd04f907700 10 osd.0 21 do_waiters -- finish -9020> 2014-05-19 18:35:33.482896 7fd03a6aa700 20 OSD::recovery_tp worker waiting -9019> 2014-05-19 18:35:33.560608 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:33.560399) v2 ==== 47+0+0 (992354158 0 0) 0x5deaa80 con 0x4cb8160 -9018> 2014-05-19 18:35:33.560624 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:33.560399) v2 ==== 47+0+0 (992354158 0 0) 0x62e0540 con 0x4cb82c0 -9017> 2014-05-19 18:35:33.560665 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.560399) v2 -- ?+0 0x662b180 con 0x4cb8160 -9016> 2014-05-19 18:35:33.560700 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9015> 2014-05-19 18:35:33.560741 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:33.560399) v2 -- ?+0 0x70a8e00 con 0x4cb82c0 -9014> 2014-05-19 18:35:33.560770 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -9013> 2014-05-19 18:35:33.664221 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:38.748201) v2 ==== 47+0+0 (1220304937 0 0) 0x772d180 con 0x31e1600 -9012> 2014-05-19 18:35:33.664262 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:38.748201) v2 -- ?+0 0x5deaa80 con 0x31e1600 -9011> 2014-05-19 18:35:33.664337 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9010> 2014-05-19 18:35:33.664315 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:38.748201) v2 ==== 47+0+0 (1220304937 0 0) 0x55c88c0 con 0x61b0160 -9009> 2014-05-19 18:35:33.664372 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:38.748201) v2 -- ?+0 0x62e0540 con 0x61b0160 -9008> 2014-05-19 18:35:33.664424 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -9007> 2014-05-19 18:35:34.380600 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:39.464593) v2 ==== 47+0+0 (2865431591 0 0) 0x73c16c0 con 0x5b21600 -9006> 2014-05-19 18:35:34.380638 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:39.464593) v2 ==== 47+0+0 (2865431591 0 0) 0x2ee08c0 con 0x5f5b760 -9005> 2014-05-19 18:35:34.380656 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.464593) v2 -- ?+0 0x772d180 con 0x5b21600 -9004> 2014-05-19 18:35:34.380692 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9003> 2014-05-19 18:35:34.380713 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.464593) v2 -- ?+0 0x55c88c0 con 0x5f5b760 -9002> 2014-05-19 18:35:34.380749 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -9001> 2014-05-19 18:35:34.474515 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 46 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:34.474328) v2 ==== 47+0+0 (1955783513 0 0) 0x2ee0fc0 con 0x4cb86e0 -9000> 2014-05-19 18:35:34.474556 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:34.474328) v2 -- ?+0 0x73c16c0 con 0x4cb86e0 -8999> 2014-05-19 18:35:34.474547 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 46 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:34.474328) v2 ==== 47+0+0 (1955783513 0 0) 0x5e29340 con 0x4cb8420 -8998> 2014-05-19 18:35:34.474591 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8997> 2014-05-19 18:35:34.474715 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:34.474328) v2 -- ?+0 0x2ee08c0 con 0x4cb8420 -8996> 2014-05-19 18:35:34.474753 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8995> 2014-05-19 18:35:34.479723 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:37.535965) v2 ==== 47+0+0 (3761380289 0 0) 0x59bddc0 con 0x61b0420 -8994> 2014-05-19 18:35:34.479772 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:37.535965) v2 -- ?+0 0x5e29340 con 0x61b0420 -8993> 2014-05-19 18:35:34.479732 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:37.535965) v2 ==== 47+0+0 (3761380289 0 0) 0x78b9500 con 0x61ae160 -8992> 2014-05-19 18:35:34.479874 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8991> 2014-05-19 18:35:34.479902 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:37.535965) v2 -- ?+0 0x2ee0fc0 con 0x61ae160 -8990> 2014-05-19 18:35:34.479945 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8989> 2014-05-19 18:35:34.482959 7fd04f907700 5 osd.0 21 tick -8988> 2014-05-19 18:35:34.483009 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8987> 2014-05-19 18:35:34.483007 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.17 < max 0.5 = yes -8986> 2014-05-19 18:35:34.483020 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -8985> 2014-05-19 18:35:34.483023 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 177.207 < min (86400 seconds) -8984> 2014-05-19 18:35:34.483043 7fd04f907700 20 osd.0 21 sched_scrub done -8983> 2014-05-19 18:35:34.483046 7fd04f907700 10 osd.0 21 do_waiters -- start -8982> 2014-05-19 18:35:34.483048 7fd04f907700 10 osd.0 21 do_waiters -- finish -8981> 2014-05-19 18:35:34.771821 7fd043ebd700 20 OSD::op_tp worker waiting -8980> 2014-05-19 18:35:34.771974 7fd04e905700 20 KeyValueStore::op_tp worker waiting -8979> 2014-05-19 18:35:34.944150 7fd0396a8700 20 OSD::command_tp worker waiting -8978> 2014-05-19 18:35:34.980350 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:38.036540) v2 ==== 47+0+0 (3913608156 0 0) 0x59bc540 con 0x61b0420 -8977> 2014-05-19 18:35:34.980341 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:38.036540) v2 ==== 47+0+0 (3913608156 0 0) 0x78b8c40 con 0x61ae160 -8976> 2014-05-19 18:35:34.980402 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:38.036540) v2 -- ?+0 0x59bddc0 con 0x61b0420 -8975> 2014-05-19 18:35:34.980478 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8974> 2014-05-19 18:35:34.980505 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:38.036540) v2 -- ?+0 0x78b9500 con 0x61ae160 -8973> 2014-05-19 18:35:34.980547 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8972> 2014-05-19 18:35:35.112345 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:38.168565) v2 ==== 47+0+0 (168201804 0 0) 0x6aa56c0 con 0x61ae420 -8971> 2014-05-19 18:35:35.112363 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:38.168565) v2 ==== 47+0+0 (168201804 0 0) 0x5cbc000 con 0x618fe40 -8970> 2014-05-19 18:35:35.112392 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:38.168565) v2 -- ?+0 0x78b8c40 con 0x61ae420 -8969> 2014-05-19 18:35:35.112445 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8968> 2014-05-19 18:35:35.112470 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:38.168565) v2 -- ?+0 0x59bc540 con 0x618fe40 -8967> 2014-05-19 18:35:35.112510 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8966> 2014-05-19 18:35:35.261158 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:35.260993) v2 ==== 47+0+0 (1618663892 0 0) 0x5deb500 con 0x4cb8160 -8965> 2014-05-19 18:35:35.261202 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:35.260993) v2 ==== 47+0+0 (1618663892 0 0) 0x6504a80 con 0x4cb82c0 -8964> 2014-05-19 18:35:35.261230 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:35.260993) v2 -- ?+0 0x6aa56c0 con 0x4cb8160 -8963> 2014-05-19 18:35:35.261267 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8962> 2014-05-19 18:35:35.261294 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:35.260993) v2 -- ?+0 0x5cbc000 con 0x4cb82c0 -8961> 2014-05-19 18:35:35.261331 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8960> 2014-05-19 18:35:35.305721 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 47 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:40.389810) v2 ==== 47+0+0 (2074040116 0 0) 0x669ee00 con 0x30946e0 -8959> 2014-05-19 18:35:35.305801 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:40.389810) v2 -- ?+0 0x5deb500 con 0x30946e0 -8958> 2014-05-19 18:35:35.305810 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 47 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:40.389810) v2 ==== 47+0+0 (2074040116 0 0) 0x662b880 con 0x3094160 -8957> 2014-05-19 18:35:35.305843 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8956> 2014-05-19 18:35:35.305863 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:40.389810) v2 -- ?+0 0x6504a80 con 0x3094160 -8955> 2014-05-19 18:35:35.305892 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8954> 2014-05-19 18:35:35.338456 7fd041eb9700 20 OSD::op_tp worker waiting -8953> 2014-05-19 18:35:35.338476 7fd03c6ae700 20 OSD::op_tp worker waiting -8952> 2014-05-19 18:35:35.338537 7fd03e6b2700 20 OSD::op_tp worker waiting -8951> 2014-05-19 18:35:35.339105 7fd03d6b0700 20 OSD::op_tp worker waiting -8950> 2014-05-19 18:35:35.339241 7fd0436bc700 20 OSD::op_tp worker waiting -8949> 2014-05-19 18:35:35.339262 7fd03bead700 20 OSD::op_tp worker waiting -8948> 2014-05-19 18:35:35.339278 7fd03f6b4700 20 OSD::op_tp worker waiting -8947> 2014-05-19 18:35:35.352841 7fd0446be700 20 OSD::op_tp worker waiting -8946> 2014-05-19 18:35:35.352859 7fd03feb5700 20 OSD::op_tp worker waiting -8945> 2014-05-19 18:35:35.360126 7fd040eb7700 20 OSD::op_tp worker waiting -8944> 2014-05-19 18:35:35.364668 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:40.448722) v2 ==== 47+0+0 (800809095 0 0) 0x7746fc0 con 0x31e1600 -8943> 2014-05-19 18:35:35.364722 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:40.448722) v2 -- ?+0 0x669ee00 con 0x31e1600 -8942> 2014-05-19 18:35:35.364755 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8941> 2014-05-19 18:35:35.364752 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:40.448722) v2 ==== 47+0+0 (800809095 0 0) 0x6009a40 con 0x61b0160 -8940> 2014-05-19 18:35:35.364797 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:40.448722) v2 -- ?+0 0x662b880 con 0x61b0160 -8939> 2014-05-19 18:35:35.364845 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8938> 2014-05-19 18:35:35.365137 7fd0406b6700 20 OSD::op_tp worker waiting -8937> 2014-05-19 18:35:35.386838 7fd039ea9700 20 OSD::disk_tp worker waiting -8936> 2014-05-19 18:35:35.390281 7fd03aeab700 20 OSD::op_tp worker waiting -8935> 2014-05-19 18:35:35.390309 7fd03eeb3700 20 OSD::op_tp worker waiting -8934> 2014-05-19 18:35:35.408263 7fd042ebb700 20 OSD::op_tp worker waiting -8933> 2014-05-19 18:35:35.408282 7fd03deb1700 20 OSD::op_tp worker waiting -8932> 2014-05-19 18:35:35.416673 7fd0426ba700 20 OSD::op_tp worker waiting -8931> 2014-05-19 18:35:35.416673 7fd04e104700 20 KeyValueStore::op_tp worker waiting -8930> 2014-05-19 18:35:35.416705 7fd03b6ac700 20 OSD::op_tp worker waiting -8929> 2014-05-19 18:35:35.416729 7fd0416b8700 20 OSD::op_tp worker waiting -8928> 2014-05-19 18:35:35.416780 7fd03ceaf700 20 OSD::op_tp worker waiting -8927> 2014-05-19 18:35:35.483135 7fd04f907700 5 osd.0 21 tick -8926> 2014-05-19 18:35:35.483185 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8925> 2014-05-19 18:35:35.483183 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.17 < max 0.5 = yes -8924> 2014-05-19 18:35:35.483196 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -8923> 2014-05-19 18:35:35.483199 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 178.207 < min (86400 seconds) -8922> 2014-05-19 18:35:35.483218 7fd04f907700 20 osd.0 21 sched_scrub done -8921> 2014-05-19 18:35:35.483221 7fd04f907700 10 osd.0 21 do_waiters -- start -8920> 2014-05-19 18:35:35.483223 7fd04f907700 10 osd.0 21 do_waiters -- finish -8919> 2014-05-19 18:35:35.865218 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:40.949256) v2 ==== 47+0+0 (2614867506 0 0) 0x55c2380 con 0x31e1600 -8918> 2014-05-19 18:35:35.865231 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:40.949256) v2 ==== 47+0+0 (2614867506 0 0) 0x6009500 con 0x61b0160 -8917> 2014-05-19 18:35:35.865268 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:40.949256) v2 -- ?+0 0x7746fc0 con 0x31e1600 -8916> 2014-05-19 18:35:35.865297 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8915> 2014-05-19 18:35:35.865347 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:40.949256) v2 -- ?+0 0x6009a40 con 0x61b0160 -8914> 2014-05-19 18:35:35.865382 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8913> 2014-05-19 18:35:36.245986 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -8912> 2014-05-19 18:35:36.246005 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -8911> 2014-05-19 18:35:36.246013 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x66f9dc0 con 0x3622580 -8910> 2014-05-19 18:35:36.246039 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x788a700 con 0x3622840 -8909> 2014-05-19 18:35:36.246076 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x6009180 con 0x3622b00 -8908> 2014-05-19 18:35:36.246091 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x7a5f340 con 0x3622dc0 -8907> 2014-05-19 18:35:36.246104 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x711e700 con 0x30594a0 -8906> 2014-05-19 18:35:36.246178 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x65421c0 con 0x3622f20 -8905> 2014-05-19 18:35:36.246210 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x6e2a380 con 0x5e7ef20 -8904> 2014-05-19 18:35:36.246235 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x682fdc0 con 0x5e7f1e0 -8903> 2014-05-19 18:35:36.246259 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x798e700 con 0x5e7e2c0 -8902> 2014-05-19 18:35:36.246292 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x5f7b340 con 0x5d7bb80 -8901> 2014-05-19 18:35:36.246320 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x73c2000 con 0x5e7ec60 -8900> 2014-05-19 18:35:36.246355 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x5e28e00 con 0x531b760 -8899> 2014-05-19 18:35:36.246370 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x5667a40 con 0x53bba20 -8898> 2014-05-19 18:35:36.246383 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x5fc3180 con 0x51cac60 -8897> 2014-05-19 18:35:36.246397 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x66f8700 con 0x5bd66e0 -8896> 2014-05-19 18:35:36.246420 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:36.246011) v2 -- ?+0 0x788a540 con 0x5bd6840 -8895> 2014-05-19 18:35:36.246418 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x711fa40 con 0x3622580 -8894> 2014-05-19 18:35:36.246463 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8893> 2014-05-19 18:35:36.246480 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x798e540 con 0x3622840 -8892> 2014-05-19 18:35:36.246501 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8891> 2014-05-19 18:35:36.246514 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x5ccc1c0 con 0x3622b00 -8890> 2014-05-19 18:35:36.246546 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8889> 2014-05-19 18:35:36.246561 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x6d61500 con 0x3622dc0 -8888> 2014-05-19 18:35:36.246576 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8887> 2014-05-19 18:35:36.246582 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x5f0a1c0 con 0x5e7ef20 -8886> 2014-05-19 18:35:36.246597 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8885> 2014-05-19 18:35:36.246602 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x6543c00 con 0x3622f20 -8884> 2014-05-19 18:35:36.246616 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8883> 2014-05-19 18:35:36.246642 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x6e2a700 con 0x5e7f1e0 -8882> 2014-05-19 18:35:36.246662 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8881> 2014-05-19 18:35:36.246670 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x725d880 con 0x30594a0 -8880> 2014-05-19 18:35:36.246723 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8879> 2014-05-19 18:35:36.246743 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x55c21c0 con 0x5e7e2c0 -8878> 2014-05-19 18:35:36.246767 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8877> 2014-05-19 18:35:36.246776 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x5519340 con 0x5d7bb80 -8876> 2014-05-19 18:35:36.246791 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8875> 2014-05-19 18:35:36.246799 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x6c4cc40 con 0x5e7ec60 -8874> 2014-05-19 18:35:36.246813 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8873> 2014-05-19 18:35:36.246818 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x73c2380 con 0x5bd6840 -8872> 2014-05-19 18:35:36.246838 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8871> 2014-05-19 18:35:36.246878 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x66f88c0 con 0x531b760 -8870> 2014-05-19 18:35:36.246912 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8869> 2014-05-19 18:35:36.246919 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x79e68c0 con 0x51cac60 -8868> 2014-05-19 18:35:36.246940 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8867> 2014-05-19 18:35:36.246949 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 52 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x72e5dc0 con 0x5bd66e0 -8866> 2014-05-19 18:35:36.246969 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8865> 2014-05-19 18:35:36.246975 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:36.246011) v2 ==== 47+0+0 (691861023 0 0) 0x60e0700 con 0x53bba20 -8864> 2014-05-19 18:35:36.246990 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8863> 2014-05-19 18:35:36.483295 7fd04f907700 5 osd.0 21 tick -8862> 2014-05-19 18:35:36.483319 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8861> 2014-05-19 18:35:36.483325 7fd04f907700 10 osd.0 21 do_waiters -- start -8860> 2014-05-19 18:35:36.483330 7fd04f907700 10 osd.0 21 do_waiters -- finish -8859> 2014-05-19 18:35:36.483356 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8858> 2014-05-19 18:35:36.550423 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:39.606651) v2 ==== 47+0+0 (2240204475 0 0) 0x3438fc0 con 0x4cb9340 -8857> 2014-05-19 18:35:36.550430 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:39.606651) v2 ==== 47+0+0 (2240204475 0 0) 0x55c9a40 con 0x4cb8dc0 -8856> 2014-05-19 18:35:36.550485 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.606651) v2 -- ?+0 0x6009500 con 0x4cb9340 -8855> 2014-05-19 18:35:36.550514 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8854> 2014-05-19 18:35:36.550531 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.606651) v2 -- ?+0 0x55c2380 con 0x4cb8dc0 -8853> 2014-05-19 18:35:36.550614 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8852> 2014-05-19 18:35:36.771917 7fd043ebd700 20 OSD::op_tp worker waiting -8851> 2014-05-19 18:35:36.772042 7fd04e905700 20 KeyValueStore::op_tp worker waiting -8850> 2014-05-19 18:35:36.944236 7fd0396a8700 20 OSD::command_tp worker waiting -8849> 2014-05-19 18:35:37.338526 7fd03c6ae700 20 OSD::op_tp worker waiting -8848> 2014-05-19 18:35:37.338552 7fd041eb9700 20 OSD::op_tp worker waiting -8847> 2014-05-19 18:35:37.338616 7fd03e6b2700 20 OSD::op_tp worker waiting -8846> 2014-05-19 18:35:37.339175 7fd03d6b0700 20 OSD::op_tp worker waiting -8845> 2014-05-19 18:35:37.339311 7fd0436bc700 20 OSD::op_tp worker waiting -8844> 2014-05-19 18:35:37.339339 7fd03bead700 20 OSD::op_tp worker waiting -8843> 2014-05-19 18:35:37.339357 7fd03f6b4700 20 OSD::op_tp worker waiting -8842> 2014-05-19 18:35:37.352941 7fd0446be700 20 OSD::op_tp worker waiting -8841> 2014-05-19 18:35:37.353000 7fd03feb5700 20 OSD::op_tp worker waiting -8840> 2014-05-19 18:35:37.360181 7fd040eb7700 20 OSD::op_tp worker waiting -8839> 2014-05-19 18:35:37.365220 7fd0406b6700 20 OSD::op_tp worker waiting -8838> 2014-05-19 18:35:37.375096 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 47 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:37.374896) v2 ==== 47+0+0 (642587656 0 0) 0x5e28fc0 con 0x4cb8420 -8837> 2014-05-19 18:35:37.375105 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 47 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:37.374896) v2 ==== 47+0+0 (642587656 0 0) 0x788b500 con 0x4cb86e0 -8836> 2014-05-19 18:35:37.375148 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:37.374896) v2 -- ?+0 0x3438fc0 con 0x4cb8420 -8835> 2014-05-19 18:35:37.375184 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8834> 2014-05-19 18:35:37.375208 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:37.374896) v2 -- ?+0 0x55c9a40 con 0x4cb86e0 -8833> 2014-05-19 18:35:37.375266 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8832> 2014-05-19 18:35:37.386919 7fd039ea9700 20 OSD::disk_tp worker waiting -8831> 2014-05-19 18:35:37.390352 7fd03aeab700 20 OSD::op_tp worker waiting -8830> 2014-05-19 18:35:37.390389 7fd03eeb3700 20 OSD::op_tp worker waiting -8829> 2014-05-19 18:35:37.408333 7fd042ebb700 20 OSD::op_tp worker waiting -8828> 2014-05-19 18:35:37.408358 7fd03deb1700 20 OSD::op_tp worker waiting -8827> 2014-05-19 18:35:37.416750 7fd0426ba700 20 OSD::op_tp worker waiting -8826> 2014-05-19 18:35:37.416763 7fd04e104700 20 KeyValueStore::op_tp worker waiting -8825> 2014-05-19 18:35:37.416802 7fd0416b8700 20 OSD::op_tp worker waiting -8824> 2014-05-19 18:35:37.416811 7fd03b6ac700 20 OSD::op_tp worker waiting -8823> 2014-05-19 18:35:37.416854 7fd03ceaf700 20 OSD::op_tp worker waiting -8822> 2014-05-19 18:35:37.483401 7fd04f907700 5 osd.0 21 tick -8821> 2014-05-19 18:35:37.483418 7fd04f907700 7 osd.0 21 do_mon_report -8820> 2014-05-19 18:35:37.483422 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -8819> 2014-05-19 18:35:37.483426 7fd04f907700 20 osd.0 21 send_pg_stats -8818> 2014-05-19 18:35:37.483433 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8817> 2014-05-19 18:35:37.483438 7fd04f907700 10 osd.0 21 do_waiters -- start -8816> 2014-05-19 18:35:37.483440 7fd04f907700 10 osd.0 21 do_waiters -- finish -8815> 2014-05-19 18:35:37.483478 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8814> 2014-05-19 18:35:38.012833 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:41.069075) v2 ==== 47+0+0 (2538897799 0 0) 0x5cbcfc0 con 0x618fe40 -8813> 2014-05-19 18:35:38.012835 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:41.069075) v2 ==== 47+0+0 (2538897799 0 0) 0x6aa4700 con 0x61ae420 -8812> 2014-05-19 18:35:38.012897 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:41.069075) v2 -- ?+0 0x5e28fc0 con 0x618fe40 -8811> 2014-05-19 18:35:38.012924 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8810> 2014-05-19 18:35:38.012947 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:41.069075) v2 -- ?+0 0x788b500 con 0x61ae420 -8809> 2014-05-19 18:35:38.012976 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8808> 2014-05-19 18:35:38.161746 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:38.161544) v2 ==== 47+0+0 (1468284538 0 0) 0x6505500 con 0x4cb82c0 -8807> 2014-05-19 18:35:38.161792 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:38.161544) v2 -- ?+0 0x5cbcfc0 con 0x4cb82c0 -8806> 2014-05-19 18:35:38.161826 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8805> 2014-05-19 18:35:38.161709 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:38.161544) v2 ==== 47+0+0 (1468284538 0 0) 0x66f8000 con 0x4cb8160 -8804> 2014-05-19 18:35:38.161855 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:38.161544) v2 -- ?+0 0x6aa4700 con 0x4cb8160 -8803> 2014-05-19 18:35:38.161876 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8802> 2014-05-19 18:35:38.251038 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:41.307263) v2 ==== 47+0+0 (4201648469 0 0) 0x5f32380 con 0x4cb8dc0 -8801> 2014-05-19 18:35:38.251066 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:41.307263) v2 ==== 47+0+0 (4201648469 0 0) 0x74a6700 con 0x4cb9340 -8800> 2014-05-19 18:35:38.251075 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:41.307263) v2 -- ?+0 0x66f8000 con 0x4cb8dc0 -8799> 2014-05-19 18:35:38.251162 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8798> 2014-05-19 18:35:38.251193 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:41.307263) v2 -- ?+0 0x6505500 con 0x4cb9340 -8797> 2014-05-19 18:35:38.251227 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8796> 2014-05-19 18:35:38.481015 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:41.537252) v2 ==== 47+0+0 (2535822098 0 0) 0x78b9a40 con 0x61ae160 -8795> 2014-05-19 18:35:38.481020 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:41.537252) v2 ==== 47+0+0 (2535822098 0 0) 0x6113180 con 0x61b0420 -8794> 2014-05-19 18:35:38.481088 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:41.537252) v2 -- ?+0 0x5f32380 con 0x61ae160 -8793> 2014-05-19 18:35:38.481122 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8792> 2014-05-19 18:35:38.481163 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:41.537252) v2 -- ?+0 0x74a6700 con 0x61b0420 -8791> 2014-05-19 18:35:38.481206 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8790> 2014-05-19 18:35:38.483520 7fd04f907700 5 osd.0 21 tick -8789> 2014-05-19 18:35:38.483540 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8788> 2014-05-19 18:35:38.483545 7fd04f907700 10 osd.0 21 do_waiters -- start -8787> 2014-05-19 18:35:38.483547 7fd04f907700 10 osd.0 21 do_waiters -- finish -8786> 2014-05-19 18:35:38.483588 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8785> 2014-05-19 18:35:38.772014 7fd043ebd700 20 OSD::op_tp worker waiting -8784> 2014-05-19 18:35:38.772118 7fd04e905700 20 KeyValueStore::op_tp worker waiting -8783> 2014-05-19 18:35:38.944308 7fd0396a8700 20 OSD::command_tp worker waiting -8782> 2014-05-19 18:35:39.075594 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 48 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:39.075409) v2 ==== 47+0+0 (1572685880 0 0) 0x5519500 con 0x4cb8420 -8781> 2014-05-19 18:35:39.075583 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 48 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:39.075409) v2 ==== 47+0+0 (1572685880 0 0) 0x6b71a40 con 0x4cb86e0 -8780> 2014-05-19 18:35:39.075644 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.075409) v2 -- ?+0 0x6113180 con 0x4cb8420 -8779> 2014-05-19 18:35:39.075679 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8778> 2014-05-19 18:35:39.075741 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.075409) v2 -- ?+0 0x78b9a40 con 0x4cb86e0 -8777> 2014-05-19 18:35:39.075784 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8776> 2014-05-19 18:35:39.146550 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -8775> 2014-05-19 18:35:39.146569 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -8774> 2014-05-19 18:35:39.146576 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x55c28c0 con 0x3622580 -8773> 2014-05-19 18:35:39.146602 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x653efc0 con 0x3622840 -8772> 2014-05-19 18:35:39.146640 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x6543500 con 0x3622b00 -8771> 2014-05-19 18:35:39.146656 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x79716c0 con 0x3622dc0 -8770> 2014-05-19 18:35:39.146670 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x60088c0 con 0x30594a0 -8769> 2014-05-19 18:35:39.146684 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x6c4c700 con 0x3622f20 -8768> 2014-05-19 18:35:39.146707 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x6b71c00 con 0x5e7ef20 -8767> 2014-05-19 18:35:39.146790 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x5f7b180 con 0x5e7f1e0 -8766> 2014-05-19 18:35:39.146827 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x7998c40 con 0x5e7e2c0 -8765> 2014-05-19 18:35:39.146857 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x511ddc0 con 0x5d7bb80 -8764> 2014-05-19 18:35:39.146885 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x72e4fc0 con 0x5e7ec60 -8763> 2014-05-19 18:35:39.146909 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x5e288c0 con 0x531b760 -8762> 2014-05-19 18:35:39.146904 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x6fe7880 con 0x3622840 -8761> 2014-05-19 18:35:39.146934 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x691b6c0 con 0x53bba20 -8760> 2014-05-19 18:35:39.146975 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x77e81c0 con 0x51cac60 -8759> 2014-05-19 18:35:39.146992 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x66f9c00 con 0x5bd66e0 -8758> 2014-05-19 18:35:39.147019 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:39.146575) v2 -- ?+0 0x788a000 con 0x5bd6840 -8757> 2014-05-19 18:35:39.147092 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8756> 2014-05-19 18:35:39.147112 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x798e380 con 0x3622b00 -8755> 2014-05-19 18:35:39.147141 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8754> 2014-05-19 18:35:39.147155 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x77461c0 con 0x3622dc0 -8753> 2014-05-19 18:35:39.147203 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8752> 2014-05-19 18:35:39.147242 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x711e8c0 con 0x3622580 -8751> 2014-05-19 18:35:39.147284 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8750> 2014-05-19 18:35:39.147298 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x8388540 con 0x30594a0 -8749> 2014-05-19 18:35:39.147328 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8748> 2014-05-19 18:35:39.147341 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x6543880 con 0x3622f20 -8747> 2014-05-19 18:35:39.147367 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8746> 2014-05-19 18:35:39.147378 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x6c4a700 con 0x5e7ef20 -8745> 2014-05-19 18:35:39.147432 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8744> 2014-05-19 18:35:39.147450 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x7a5e8c0 con 0x5e7f1e0 -8743> 2014-05-19 18:35:39.147484 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8742> 2014-05-19 18:35:39.147499 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x3510380 con 0x5e7e2c0 -8741> 2014-05-19 18:35:39.147530 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8740> 2014-05-19 18:35:39.147543 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x55181c0 con 0x5d7bb80 -8739> 2014-05-19 18:35:39.147570 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8738> 2014-05-19 18:35:39.147588 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x6c4c000 con 0x5e7ec60 -8737> 2014-05-19 18:35:39.147618 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8736> 2014-05-19 18:35:39.147632 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x692d340 con 0x531b760 -8735> 2014-05-19 18:35:39.147660 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8734> 2014-05-19 18:35:39.147677 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x6378a80 con 0x53bba20 -8733> 2014-05-19 18:35:39.147706 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8732> 2014-05-19 18:35:39.147719 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x8389880 con 0x51cac60 -8731> 2014-05-19 18:35:39.147749 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8730> 2014-05-19 18:35:39.147763 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x72e4540 con 0x5bd66e0 -8729> 2014-05-19 18:35:39.147793 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8728> 2014-05-19 18:35:39.147806 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 53 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:39.146575) v2 ==== 47+0+0 (1492028796 0 0) 0x73c2540 con 0x5bd6840 -8727> 2014-05-19 18:35:39.147834 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8726> 2014-05-19 18:35:39.338622 7fd03c6ae700 20 OSD::op_tp worker waiting -8725> 2014-05-19 18:35:39.338645 7fd041eb9700 20 OSD::op_tp worker waiting -8724> 2014-05-19 18:35:39.338697 7fd03e6b2700 20 OSD::op_tp worker waiting -8723> 2014-05-19 18:35:39.339239 7fd03d6b0700 20 OSD::op_tp worker waiting -8722> 2014-05-19 18:35:39.339383 7fd0436bc700 20 OSD::op_tp worker waiting -8721> 2014-05-19 18:35:39.339415 7fd03bead700 20 OSD::op_tp worker waiting -8720> 2014-05-19 18:35:39.339438 7fd03f6b4700 20 OSD::op_tp worker waiting -8719> 2014-05-19 18:35:39.353042 7fd0446be700 20 OSD::op_tp worker waiting -8718> 2014-05-19 18:35:39.353078 7fd03feb5700 20 OSD::op_tp worker waiting -8717> 2014-05-19 18:35:39.360274 7fd040eb7700 20 OSD::op_tp worker waiting -8716> 2014-05-19 18:35:39.365300 7fd0406b6700 20 OSD::op_tp worker waiting -8715> 2014-05-19 18:35:39.365543 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:44.449715) v2 ==== 47+0+0 (1070824804 0 0) 0x2ee16c0 con 0x31e1600 -8714> 2014-05-19 18:35:39.365536 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:44.449715) v2 ==== 47+0+0 (1070824804 0 0) 0x72e4c40 con 0x61b0160 -8713> 2014-05-19 18:35:39.365592 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.449715) v2 -- ?+0 0x6b71a40 con 0x31e1600 -8712> 2014-05-19 18:35:39.365663 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8711> 2014-05-19 18:35:39.365686 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.449715) v2 -- ?+0 0x5519500 con 0x61b0160 -8710> 2014-05-19 18:35:39.365728 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8709> 2014-05-19 18:35:39.387016 7fd039ea9700 20 OSD::disk_tp worker waiting -8708> 2014-05-19 18:35:39.390446 7fd03aeab700 20 OSD::op_tp worker waiting -8707> 2014-05-19 18:35:39.390468 7fd03eeb3700 20 OSD::op_tp worker waiting -8706> 2014-05-19 18:35:39.408412 7fd042ebb700 20 OSD::op_tp worker waiting -8705> 2014-05-19 18:35:39.408441 7fd03deb1700 20 OSD::op_tp worker waiting -8704> 2014-05-19 18:35:39.416841 7fd04e104700 20 KeyValueStore::op_tp worker waiting -8703> 2014-05-19 18:35:39.416846 7fd0426ba700 20 OSD::op_tp worker waiting -8702> 2014-05-19 18:35:39.416878 7fd0416b8700 20 OSD::op_tp worker waiting -8701> 2014-05-19 18:35:39.416907 7fd03b6ac700 20 OSD::op_tp worker waiting -8700> 2014-05-19 18:35:39.416931 7fd03ceaf700 20 OSD::op_tp worker waiting -8699> 2014-05-19 18:35:39.483621 7fd04f907700 5 osd.0 21 tick -8698> 2014-05-19 18:35:39.483642 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8697> 2014-05-19 18:35:39.483646 7fd04f907700 10 osd.0 21 do_waiters -- start -8696> 2014-05-19 18:35:39.483649 7fd04f907700 10 osd.0 21 do_waiters -- finish -8695> 2014-05-19 18:35:39.483690 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8694> 2014-05-19 18:35:39.906484 7fd0456c0700 10 monclient: tick -8693> 2014-05-19 18:35:39.906497 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:35:09.906496) -8692> 2014-05-19 18:35:39.906518 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:35:39.906518; renew after: 2014-05-19 18:35:39.994233) -- no -8691> 2014-05-19 18:35:40.181679 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:43.237938) v2 ==== 47+0+0 (4122727371 0 0) 0x6113dc0 con 0x61b0420 -8690> 2014-05-19 18:35:40.181705 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:43.237938) v2 ==== 47+0+0 (4122727371 0 0) 0x79801c0 con 0x61ae160 -8689> 2014-05-19 18:35:40.181735 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:43.237938) v2 -- ?+0 0x72e4c40 con 0x61b0420 -8688> 2014-05-19 18:35:40.181830 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8687> 2014-05-19 18:35:40.181885 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:43.237938) v2 -- ?+0 0x2ee16c0 con 0x61ae160 -8686> 2014-05-19 18:35:40.181922 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8685> 2014-05-19 18:35:40.280928 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:45.365133) v2 ==== 47+0+0 (2102799038 0 0) 0x77e8e00 con 0x5b21600 -8684> 2014-05-19 18:35:40.280946 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:45.365133) v2 ==== 47+0+0 (2102799038 0 0) 0x5f0aa80 con 0x5f5b760 -8683> 2014-05-19 18:35:40.280978 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:45.365133) v2 -- ?+0 0x79801c0 con 0x5b21600 -8682> 2014-05-19 18:35:40.281019 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8681> 2014-05-19 18:35:40.281043 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:45.365133) v2 -- ?+0 0x6113dc0 con 0x5f5b760 -8680> 2014-05-19 18:35:40.281082 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8679> 2014-05-19 18:35:40.483716 7fd04f907700 5 osd.0 21 tick -8678> 2014-05-19 18:35:40.483764 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.15 < max 0.5 = yes -8677> 2014-05-19 18:35:40.483777 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -8676> 2014-05-19 18:35:40.483773 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8675> 2014-05-19 18:35:40.483780 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 183.208 < min (86400 seconds) -8674> 2014-05-19 18:35:40.483799 7fd04f907700 20 osd.0 21 sched_scrub done -8673> 2014-05-19 18:35:40.483802 7fd04f907700 10 osd.0 21 do_waiters -- start -8672> 2014-05-19 18:35:40.483803 7fd04f907700 10 osd.0 21 do_waiters -- finish -8671> 2014-05-19 18:35:40.772111 7fd043ebd700 20 OSD::op_tp worker waiting -8670> 2014-05-19 18:35:40.772196 7fd04e905700 20 KeyValueStore::op_tp worker waiting -8669> 2014-05-19 18:35:40.913385 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:43.969637) v2 ==== 47+0+0 (1882438253 0 0) 0x6aa5500 con 0x61ae420 -8668> 2014-05-19 18:35:40.913386 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:43.969637) v2 ==== 47+0+0 (1882438253 0 0) 0x7a36fc0 con 0x618fe40 -8667> 2014-05-19 18:35:40.913434 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:43.969637) v2 -- ?+0 0x77e8e00 con 0x61ae420 -8666> 2014-05-19 18:35:40.913479 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8665> 2014-05-19 18:35:40.913535 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:43.969637) v2 -- ?+0 0x5f0aa80 con 0x618fe40 -8664> 2014-05-19 18:35:40.913575 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8663> 2014-05-19 18:35:40.944387 7fd0396a8700 20 OSD::command_tp worker waiting -8662> 2014-05-19 18:35:41.062335 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:41.062126) v2 ==== 47+0+0 (173701424 0 0) 0x7930700 con 0x4cb82c0 -8661> 2014-05-19 18:35:41.062381 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:41.062126) v2 -- ?+0 0x7a36fc0 con 0x4cb82c0 -8660> 2014-05-19 18:35:41.062408 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8659> 2014-05-19 18:35:41.062394 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:41.062126) v2 ==== 47+0+0 (173701424 0 0) 0x66f8380 con 0x4cb8160 -8658> 2014-05-19 18:35:41.062494 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:41.062126) v2 -- ?+0 0x6aa5500 con 0x4cb8160 -8657> 2014-05-19 18:35:41.062531 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8656> 2014-05-19 18:35:41.065901 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:46.150180) v2 ==== 47+0+0 (3362032179 0 0) 0x6aa4700 con 0x31e1600 -8655> 2014-05-19 18:35:41.065933 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.150180) v2 -- ?+0 0x66f8380 con 0x31e1600 -8654> 2014-05-19 18:35:41.065968 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8653> 2014-05-19 18:35:41.066004 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:46.150180) v2 ==== 47+0+0 (3362032179 0 0) 0x72e5c00 con 0x61b0160 -8652> 2014-05-19 18:35:41.066041 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.150180) v2 -- ?+0 0x7930700 con 0x61b0160 -8651> 2014-05-19 18:35:41.066072 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8650> 2014-05-19 18:35:41.151608 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:44.207822) v2 ==== 47+0+0 (2639048891 0 0) 0x5497dc0 con 0x4cb9340 -8649> 2014-05-19 18:35:41.151598 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:44.207822) v2 ==== 47+0+0 (2639048891 0 0) 0x5e29340 con 0x4cb8dc0 -8648> 2014-05-19 18:35:41.151656 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.207822) v2 -- ?+0 0x72e5c00 con 0x4cb9340 -8647> 2014-05-19 18:35:41.151687 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8646> 2014-05-19 18:35:41.151762 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.207822) v2 -- ?+0 0x6aa4700 con 0x4cb8dc0 -8645> 2014-05-19 18:35:41.151795 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8644> 2014-05-19 18:35:41.206134 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 48 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:46.290349) v2 ==== 47+0+0 (1360904517 0 0) 0x662a1c0 con 0x3094160 -8643> 2014-05-19 18:35:41.206123 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 48 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:46.290349) v2 ==== 47+0+0 (1360904517 0 0) 0x77e96c0 con 0x30946e0 -8642> 2014-05-19 18:35:41.206188 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.290349) v2 -- ?+0 0x5497dc0 con 0x3094160 -8641> 2014-05-19 18:35:41.206234 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8640> 2014-05-19 18:35:41.206266 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.290349) v2 -- ?+0 0x5e29340 con 0x30946e0 -8639> 2014-05-19 18:35:41.206302 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8638> 2014-05-19 18:35:41.338718 7fd03c6ae700 20 OSD::op_tp worker waiting -8637> 2014-05-19 18:35:41.338743 7fd041eb9700 20 OSD::op_tp worker waiting -8636> 2014-05-19 18:35:41.338774 7fd03e6b2700 20 OSD::op_tp worker waiting -8635> 2014-05-19 18:35:41.339304 7fd03d6b0700 20 OSD::op_tp worker waiting -8634> 2014-05-19 18:35:41.339461 7fd0436bc700 20 OSD::op_tp worker waiting -8633> 2014-05-19 18:35:41.339490 7fd03bead700 20 OSD::op_tp worker waiting -8632> 2014-05-19 18:35:41.339509 7fd03f6b4700 20 OSD::op_tp worker waiting -8631> 2014-05-19 18:35:41.353123 7fd0446be700 20 OSD::op_tp worker waiting -8630> 2014-05-19 18:35:41.353156 7fd03feb5700 20 OSD::op_tp worker waiting -8629> 2014-05-19 18:35:41.360353 7fd040eb7700 20 OSD::op_tp worker waiting -8628> 2014-05-19 18:35:41.365398 7fd0406b6700 20 OSD::op_tp worker waiting -8627> 2014-05-19 18:35:41.387111 7fd039ea9700 20 OSD::disk_tp worker waiting -8626> 2014-05-19 18:35:41.390546 7fd03aeab700 20 OSD::op_tp worker waiting -8625> 2014-05-19 18:35:41.390572 7fd03eeb3700 20 OSD::op_tp worker waiting -8624> 2014-05-19 18:35:41.408498 7fd042ebb700 20 OSD::op_tp worker waiting -8623> 2014-05-19 18:35:41.408520 7fd03deb1700 20 OSD::op_tp worker waiting -8622> 2014-05-19 18:35:41.416952 7fd0426ba700 20 OSD::op_tp worker waiting -8621> 2014-05-19 18:35:41.416953 7fd04e104700 20 KeyValueStore::op_tp worker waiting -8620> 2014-05-19 18:35:41.416984 7fd03b6ac700 20 OSD::op_tp worker waiting -8619> 2014-05-19 18:35:41.417018 7fd0416b8700 20 OSD::op_tp worker waiting -8618> 2014-05-19 18:35:41.417043 7fd03ceaf700 20 OSD::op_tp worker waiting -8617> 2014-05-19 18:35:41.483892 7fd04f907700 5 osd.0 21 tick -8616> 2014-05-19 18:35:41.483912 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8615> 2014-05-19 18:35:41.483917 7fd04f907700 10 osd.0 21 do_waiters -- start -8614> 2014-05-19 18:35:41.483919 7fd04f907700 10 osd.0 21 do_waiters -- finish -8613> 2014-05-19 18:35:41.483960 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8612> 2014-05-19 18:35:42.484004 7fd04f907700 5 osd.0 21 tick -8611> 2014-05-19 18:35:42.484021 7fd04f907700 7 osd.0 21 do_mon_report -8610> 2014-05-19 18:35:42.484025 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -8609> 2014-05-19 18:35:42.484028 7fd04f907700 20 osd.0 21 send_pg_stats -8608> 2014-05-19 18:35:42.484036 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8607> 2014-05-19 18:35:42.484040 7fd04f907700 10 osd.0 21 do_waiters -- start -8606> 2014-05-19 18:35:42.484042 7fd04f907700 10 osd.0 21 do_waiters -- finish -8605> 2014-05-19 18:35:42.484081 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8604> 2014-05-19 18:35:42.772201 7fd043ebd700 20 OSD::op_tp worker waiting -8603> 2014-05-19 18:35:42.772257 7fd04e905700 20 KeyValueStore::op_tp worker waiting -8602> 2014-05-19 18:35:42.944460 7fd0396a8700 20 OSD::command_tp worker waiting -8601> 2014-05-19 18:35:43.082453 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:46.138773) v2 ==== 47+0+0 (1445634734 0 0) 0x6112e00 con 0x61b0420 -8600> 2014-05-19 18:35:43.082488 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:46.138773) v2 ==== 47+0+0 (1445634734 0 0) 0x55f5c00 con 0x61ae160 -8599> 2014-05-19 18:35:43.082508 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.138773) v2 -- ?+0 0x662a1c0 con 0x61b0420 -8598> 2014-05-19 18:35:43.082626 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8597> 2014-05-19 18:35:43.082664 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.138773) v2 -- ?+0 0x77e96c0 con 0x61ae160 -8596> 2014-05-19 18:35:43.082721 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8595> 2014-05-19 18:35:43.338817 7fd03c6ae700 20 OSD::op_tp worker waiting -8594> 2014-05-19 18:35:43.338841 7fd041eb9700 20 OSD::op_tp worker waiting -8593> 2014-05-19 18:35:43.338861 7fd03e6b2700 20 OSD::op_tp worker waiting -8592> 2014-05-19 18:35:43.339367 7fd03d6b0700 20 OSD::op_tp worker waiting -8591> 2014-05-19 18:35:43.339543 7fd0436bc700 20 OSD::op_tp worker waiting -8590> 2014-05-19 18:35:43.339578 7fd03bead700 20 OSD::op_tp worker waiting -8589> 2014-05-19 18:35:43.339599 7fd03f6b4700 20 OSD::op_tp worker waiting -8588> 2014-05-19 18:35:43.353221 7fd0446be700 20 OSD::op_tp worker waiting -8587> 2014-05-19 18:35:43.353246 7fd03feb5700 20 OSD::op_tp worker waiting -8586> 2014-05-19 18:35:43.360434 7fd040eb7700 20 OSD::op_tp worker waiting -8585> 2014-05-19 18:35:43.365495 7fd0406b6700 20 OSD::op_tp worker waiting -8584> 2014-05-19 18:35:43.387197 7fd039ea9700 20 OSD::disk_tp worker waiting -8583> 2014-05-19 18:35:43.390645 7fd03aeab700 20 OSD::op_tp worker waiting -8582> 2014-05-19 18:35:43.390670 7fd03eeb3700 20 OSD::op_tp worker waiting -8581> 2014-05-19 18:35:43.408598 7fd03deb1700 20 OSD::op_tp worker waiting -8580> 2014-05-19 18:35:43.408623 7fd042ebb700 20 OSD::op_tp worker waiting -8579> 2014-05-19 18:35:43.417058 7fd04e104700 20 KeyValueStore::op_tp worker waiting -8578> 2014-05-19 18:35:43.417060 7fd0426ba700 20 OSD::op_tp worker waiting -8577> 2014-05-19 18:35:43.417095 7fd03b6ac700 20 OSD::op_tp worker waiting -8576> 2014-05-19 18:35:43.417109 7fd03ceaf700 20 OSD::op_tp worker waiting -8575> 2014-05-19 18:35:43.417125 7fd0416b8700 20 OSD::op_tp worker waiting -8574> 2014-05-19 18:35:43.484128 7fd04f907700 5 osd.0 21 tick -8573> 2014-05-19 18:35:43.484149 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8572> 2014-05-19 18:35:43.484154 7fd04f907700 10 osd.0 21 do_waiters -- start -8571> 2014-05-19 18:35:43.484156 7fd04f907700 10 osd.0 21 do_waiters -- finish -8570> 2014-05-19 18:35:43.484197 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8569> 2014-05-19 18:35:43.776160 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:43.775970) v2 ==== 47+0+0 (3008757273 0 0) 0x73c2700 con 0x4cb86e0 -8568> 2014-05-19 18:35:43.776169 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:43.775970) v2 ==== 47+0+0 (3008757273 0 0) 0x6113500 con 0x4cb8420 -8567> 2014-05-19 18:35:43.776201 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:43.775970) v2 -- ?+0 0x55f5c00 con 0x4cb86e0 -8566> 2014-05-19 18:35:43.776231 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8565> 2014-05-19 18:35:43.776294 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:43.775970) v2 -- ?+0 0x6112e00 con 0x4cb8420 -8564> 2014-05-19 18:35:43.776350 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8563> 2014-05-19 18:35:43.966307 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:49.050629) v2 ==== 47+0+0 (1502635371 0 0) 0x6aa56c0 con 0x31e1600 -8562> 2014-05-19 18:35:43.966332 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:49.050629) v2 ==== 47+0+0 (1502635371 0 0) 0x5ccc8c0 con 0x61b0160 -8561> 2014-05-19 18:35:43.966356 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.050629) v2 -- ?+0 0x73c2700 con 0x31e1600 -8560> 2014-05-19 18:35:43.966388 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8559> 2014-05-19 18:35:43.966412 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.050629) v2 -- ?+0 0x6113500 con 0x61b0160 -8558> 2014-05-19 18:35:43.966456 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8557> 2014-05-19 18:35:44.052048 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:47.108347) v2 ==== 47+0+0 (3666469758 0 0) 0x72e4c40 con 0x4cb8dc0 -8556> 2014-05-19 18:35:44.052088 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:47.108347) v2 ==== 47+0+0 (3666469758 0 0) 0x5497a40 con 0x4cb9340 -8555> 2014-05-19 18:35:44.052095 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:47.108347) v2 -- ?+0 0x6aa56c0 con 0x4cb8dc0 -8554> 2014-05-19 18:35:44.052144 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8553> 2014-05-19 18:35:44.052168 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:47.108347) v2 -- ?+0 0x5ccc8c0 con 0x4cb9340 -8552> 2014-05-19 18:35:44.052205 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8551> 2014-05-19 18:35:44.447161 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -8550> 2014-05-19 18:35:44.447179 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -8549> 2014-05-19 18:35:44.447187 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x653e8c0 con 0x3622580 -8548> 2014-05-19 18:35:44.447213 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x51a68c0 con 0x3622840 -8547> 2014-05-19 18:35:44.447267 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x7970e00 con 0x3622b00 -8546> 2014-05-19 18:35:44.447299 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x6008000 con 0x3622dc0 -8545> 2014-05-19 18:35:44.447329 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x6543c00 con 0x30594a0 -8544> 2014-05-19 18:35:44.447356 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x5f0a1c0 con 0x3622f20 -8543> 2014-05-19 18:35:44.447427 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x6d61500 con 0x5e7ef20 -8542> 2014-05-19 18:35:44.447462 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x5ccc1c0 con 0x5e7f1e0 -8541> 2014-05-19 18:35:44.447493 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x798e540 con 0x5e7e2c0 -8540> 2014-05-19 18:35:44.447560 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x711fa40 con 0x5d7bb80 -8539> 2014-05-19 18:35:44.447598 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x55c36c0 con 0x3622b00 -8538> 2014-05-19 18:35:44.447635 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x73c21c0 con 0x5e7ec60 -8537> 2014-05-19 18:35:44.447668 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x72e5500 con 0x531b760 -8536> 2014-05-19 18:35:44.447691 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x79e6c40 con 0x53bba20 -8535> 2014-05-19 18:35:44.447705 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x2ee01c0 con 0x51cac60 -8534> 2014-05-19 18:35:44.447721 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x66f9500 con 0x5bd66e0 -8533> 2014-05-19 18:35:44.447736 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:44.447185) v2 -- ?+0 0x612ce00 con 0x5bd6840 -8532> 2014-05-19 18:35:44.447773 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8531> 2014-05-19 18:35:44.447795 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x6fe61c0 con 0x3622dc0 -8530> 2014-05-19 18:35:44.447828 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8529> 2014-05-19 18:35:44.447861 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x7747880 con 0x3622580 -8528> 2014-05-19 18:35:44.447882 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8527> 2014-05-19 18:35:44.447893 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x6efe540 con 0x3622840 -8526> 2014-05-19 18:35:44.447908 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8525> 2014-05-19 18:35:44.447914 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x64d3180 con 0x5e7ef20 -8524> 2014-05-19 18:35:44.447943 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8523> 2014-05-19 18:35:44.447951 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x6711500 con 0x5e7e2c0 -8522> 2014-05-19 18:35:44.447967 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8521> 2014-05-19 18:35:44.447980 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x60081c0 con 0x5d7bb80 -8520> 2014-05-19 18:35:44.447997 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8519> 2014-05-19 18:35:44.448005 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x6c4ca80 con 0x30594a0 -8518> 2014-05-19 18:35:44.448021 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8517> 2014-05-19 18:35:44.448028 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x8388700 con 0x3622f20 -8516> 2014-05-19 18:35:44.448042 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8515> 2014-05-19 18:35:44.448072 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x5f06540 con 0x5e7f1e0 -8514> 2014-05-19 18:35:44.448107 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8513> 2014-05-19 18:35:44.448135 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x59bcc40 con 0x53bba20 -8512> 2014-05-19 18:35:44.448172 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8511> 2014-05-19 18:35:44.448188 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x73c3a40 con 0x5bd6840 -8510> 2014-05-19 18:35:44.448218 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8509> 2014-05-19 18:35:44.448229 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x6543a40 con 0x5e7ec60 -8508> 2014-05-19 18:35:44.448245 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8507> 2014-05-19 18:35:44.448253 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x788b180 con 0x531b760 -8506> 2014-05-19 18:35:44.448268 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8505> 2014-05-19 18:35:44.448273 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x68ab500 con 0x51cac60 -8504> 2014-05-19 18:35:44.448289 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8503> 2014-05-19 18:35:44.448297 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 54 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:44.447185) v2 ==== 47+0+0 (789768496 0 0) 0x72e48c0 con 0x5bd66e0 -8502> 2014-05-19 18:35:44.448311 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8501> 2014-05-19 18:35:44.484245 7fd04f907700 5 osd.0 21 tick -8500> 2014-05-19 18:35:44.484266 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8499> 2014-05-19 18:35:44.484270 7fd04f907700 10 osd.0 21 do_waiters -- start -8498> 2014-05-19 18:35:44.484273 7fd04f907700 10 osd.0 21 do_waiters -- finish -8497> 2014-05-19 18:35:44.484302 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8496> 2014-05-19 18:35:44.772299 7fd043ebd700 20 OSD::op_tp worker waiting -8495> 2014-05-19 18:35:44.772328 7fd04e905700 20 KeyValueStore::op_tp worker waiting -8494> 2014-05-19 18:35:44.944535 7fd0396a8700 20 OSD::command_tp worker waiting -8493> 2014-05-19 18:35:45.338879 7fd03c6ae700 20 OSD::op_tp worker waiting -8492> 2014-05-19 18:35:45.338933 7fd041eb9700 20 OSD::op_tp worker waiting -8491> 2014-05-19 18:35:45.338949 7fd03e6b2700 20 OSD::op_tp worker waiting -8490> 2014-05-19 18:35:45.339434 7fd03d6b0700 20 OSD::op_tp worker waiting -8489> 2014-05-19 18:35:45.339617 7fd0436bc700 20 OSD::op_tp worker waiting -8488> 2014-05-19 18:35:45.339675 7fd03bead700 20 OSD::op_tp worker waiting -8487> 2014-05-19 18:35:45.339706 7fd03f6b4700 20 OSD::op_tp worker waiting -8486> 2014-05-19 18:35:45.353324 7fd0446be700 20 OSD::op_tp worker waiting -8485> 2014-05-19 18:35:45.353378 7fd03feb5700 20 OSD::op_tp worker waiting -8484> 2014-05-19 18:35:45.360483 7fd040eb7700 20 OSD::op_tp worker waiting -8483> 2014-05-19 18:35:45.365591 7fd0406b6700 20 OSD::op_tp worker waiting -8482> 2014-05-19 18:35:45.387276 7fd039ea9700 20 OSD::disk_tp worker waiting -8481> 2014-05-19 18:35:45.390744 7fd03aeab700 20 OSD::op_tp worker waiting -8480> 2014-05-19 18:35:45.390768 7fd03eeb3700 20 OSD::op_tp worker waiting -8479> 2014-05-19 18:35:45.408686 7fd03deb1700 20 OSD::op_tp worker waiting -8478> 2014-05-19 18:35:45.408705 7fd042ebb700 20 OSD::op_tp worker waiting -8477> 2014-05-19 18:35:45.417163 7fd04e104700 20 KeyValueStore::op_tp worker waiting -8476> 2014-05-19 18:35:45.417168 7fd0426ba700 20 OSD::op_tp worker waiting -8475> 2014-05-19 18:35:45.417202 7fd03b6ac700 20 OSD::op_tp worker waiting -8474> 2014-05-19 18:35:45.417225 7fd03ceaf700 20 OSD::op_tp worker waiting -8473> 2014-05-19 18:35:45.417288 7fd0416b8700 20 OSD::op_tp worker waiting -8472> 2014-05-19 18:35:45.476671 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:45.476498) v2 ==== 47+0+0 (354115329 0 0) 0x7a5e000 con 0x4cb8420 -8471> 2014-05-19 18:35:45.476717 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:45.476498) v2 -- ?+0 0x5497a40 con 0x4cb8420 -8470> 2014-05-19 18:35:45.476749 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8469> 2014-05-19 18:35:45.476745 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:45.476498) v2 ==== 47+0+0 (354115329 0 0) 0x60f6000 con 0x4cb86e0 -8468> 2014-05-19 18:35:45.476793 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:45.476498) v2 -- ?+0 0x72e4c40 con 0x4cb86e0 -8467> 2014-05-19 18:35:45.476872 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8466> 2014-05-19 18:35:45.484335 7fd04f907700 5 osd.0 21 tick -8465> 2014-05-19 18:35:45.484352 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8464> 2014-05-19 18:35:45.484356 7fd04f907700 10 osd.0 21 do_waiters -- start -8463> 2014-05-19 18:35:45.484358 7fd04f907700 10 osd.0 21 do_waiters -- finish -8462> 2014-05-19 18:35:45.484358 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8461> 2014-05-19 18:35:45.983071 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:49.039436) v2 ==== 47+0+0 (3301187971 0 0) 0x5c47880 con 0x61b0420 -8460> 2014-05-19 18:35:45.983112 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.039436) v2 -- ?+0 0x7a5e000 con 0x61b0420 -8459> 2014-05-19 18:35:45.983134 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8458> 2014-05-19 18:35:45.983102 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:49.039436) v2 ==== 47+0+0 (3301187971 0 0) 0x5667340 con 0x61ae160 -8457> 2014-05-19 18:35:45.983188 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.039436) v2 -- ?+0 0x60f6000 con 0x61ae160 -8456> 2014-05-19 18:35:45.983218 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8455> 2014-05-19 18:35:46.147861 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -8454> 2014-05-19 18:35:46.147879 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -8453> 2014-05-19 18:35:46.147888 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x55c3880 con 0x3622580 -8452> 2014-05-19 18:35:46.147913 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x682e700 con 0x3622840 -8451> 2014-05-19 18:35:46.147952 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x798f880 con 0x3622b00 -8450> 2014-05-19 18:35:46.147967 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x65436c0 con 0x3622dc0 -8449> 2014-05-19 18:35:46.147999 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x725ca80 con 0x30594a0 -8448> 2014-05-19 18:35:46.148028 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x5518700 con 0x3622f20 -8447> 2014-05-19 18:35:46.148092 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x6e2ba40 con 0x5e7ef20 -8446> 2014-05-19 18:35:46.148121 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x79988c0 con 0x5e7f1e0 -8445> 2014-05-19 18:35:46.148136 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x6d60700 con 0x5e7e2c0 -8444> 2014-05-19 18:35:46.148165 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x711ee00 con 0x5d7bb80 -8443> 2014-05-19 18:35:46.148232 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x73c28c0 con 0x5e7ec60 -8442> 2014-05-19 18:35:46.148234 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x7747180 con 0x3622580 -8441> 2014-05-19 18:35:46.148264 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x72e4700 con 0x531b760 -8440> 2014-05-19 18:35:46.148291 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x662bc00 con 0x53bba20 -8439> 2014-05-19 18:35:46.148312 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x66f8e00 con 0x51cac60 -8438> 2014-05-19 18:35:46.148328 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x725da40 con 0x5bd66e0 -8437> 2014-05-19 18:35:46.148342 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:46.147886) v2 -- ?+0 0x6008380 con 0x5bd6840 -8436> 2014-05-19 18:35:46.148401 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8435> 2014-05-19 18:35:46.148430 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x55c2380 con 0x3622b00 -8434> 2014-05-19 18:35:46.148479 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8433> 2014-05-19 18:35:46.148507 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x7971dc0 con 0x3622dc0 -8432> 2014-05-19 18:35:46.148527 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8431> 2014-05-19 18:35:46.148536 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x6c4d500 con 0x30594a0 -8430> 2014-05-19 18:35:46.148552 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8429> 2014-05-19 18:35:46.148560 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x5f328c0 con 0x3622840 -8428> 2014-05-19 18:35:46.148607 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8427> 2014-05-19 18:35:46.148622 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x64d2a80 con 0x5e7ef20 -8426> 2014-05-19 18:35:46.148644 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8425> 2014-05-19 18:35:46.148650 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x612dc00 con 0x5e7e2c0 -8424> 2014-05-19 18:35:46.148704 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8423> 2014-05-19 18:35:46.148780 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x6008e00 con 0x5d7bb80 -8422> 2014-05-19 18:35:46.148810 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8421> 2014-05-19 18:35:46.148820 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x65421c0 con 0x5e7ec60 -8420> 2014-05-19 18:35:46.148838 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8419> 2014-05-19 18:35:46.148855 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x788a700 con 0x531b760 -8418> 2014-05-19 18:35:46.148873 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8417> 2014-05-19 18:35:46.148884 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x74a6700 con 0x53bba20 -8416> 2014-05-19 18:35:46.148899 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8415> 2014-05-19 18:35:46.148905 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x70a9dc0 con 0x51cac60 -8414> 2014-05-19 18:35:46.148921 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8413> 2014-05-19 18:35:46.148929 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x72e41c0 con 0x5bd66e0 -8412> 2014-05-19 18:35:46.148945 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8411> 2014-05-19 18:35:46.148952 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 55 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x60f6700 con 0x5bd6840 -8410> 2014-05-19 18:35:46.148967 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8409> 2014-05-19 18:35:46.148974 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x59bcfc0 con 0x3622f20 -8408> 2014-05-19 18:35:46.148989 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8407> 2014-05-19 18:35:46.148997 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.147886) v2 ==== 47+0+0 (599566219 0 0) 0x5f07340 con 0x5e7f1e0 -8406> 2014-05-19 18:35:46.149014 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8405> 2014-05-19 18:35:46.181230 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:51.265686) v2 ==== 47+0+0 (1024725406 0 0) 0x60e2a80 con 0x5f5b760 -8404> 2014-05-19 18:35:46.181283 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:51.265686) v2 -- ?+0 0x5c47880 con 0x5f5b760 -8403> 2014-05-19 18:35:46.181329 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8402> 2014-05-19 18:35:46.181232 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:51.265686) v2 ==== 47+0+0 (1024725406 0 0) 0x77e8c40 con 0x5b21600 -8401> 2014-05-19 18:35:46.181433 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:51.265686) v2 -- ?+0 0x5667340 con 0x5b21600 -8400> 2014-05-19 18:35:46.181473 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8399> 2014-05-19 18:35:46.213895 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:49.270219) v2 ==== 47+0+0 (1101396908 0 0) 0x7a37180 con 0x618fe40 -8398> 2014-05-19 18:35:46.213901 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:49.270219) v2 ==== 47+0+0 (1101396908 0 0) 0x6aa5880 con 0x61ae420 -8397> 2014-05-19 18:35:46.213947 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.270219) v2 -- ?+0 0x60e2a80 con 0x618fe40 -8396> 2014-05-19 18:35:46.213988 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8395> 2014-05-19 18:35:46.214011 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.270219) v2 -- ?+0 0x77e8c40 con 0x61ae420 -8394> 2014-05-19 18:35:46.214056 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8393> 2014-05-19 18:35:46.362881 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:46.362665) v2 ==== 47+0+0 (3993647129 0 0) 0x662b6c0 con 0x4cb8160 -8392> 2014-05-19 18:35:46.362897 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:46.362665) v2 ==== 47+0+0 (3993647129 0 0) 0x5cbca80 con 0x4cb82c0 -8391> 2014-05-19 18:35:46.362937 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.362665) v2 -- ?+0 0x6aa5880 con 0x4cb8160 -8390> 2014-05-19 18:35:46.362980 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8389> 2014-05-19 18:35:46.363024 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:46.362665) v2 -- ?+0 0x7a37180 con 0x4cb82c0 -8388> 2014-05-19 18:35:46.363061 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8387> 2014-05-19 18:35:46.484431 7fd04f907700 5 osd.0 21 tick -8386> 2014-05-19 18:35:46.484453 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8385> 2014-05-19 18:35:46.484457 7fd04f907700 10 osd.0 21 do_waiters -- start -8384> 2014-05-19 18:35:46.484460 7fd04f907700 10 osd.0 21 do_waiters -- finish -8383> 2014-05-19 18:35:46.484501 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8382> 2014-05-19 18:35:46.772383 7fd043ebd700 20 OSD::op_tp worker waiting -8381> 2014-05-19 18:35:46.772402 7fd04e905700 20 KeyValueStore::op_tp worker waiting -8380> 2014-05-19 18:35:46.866589 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:51.951069) v2 ==== 47+0+0 (81849990 0 0) 0x79e7340 con 0x31e1600 -8379> 2014-05-19 18:35:46.866644 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:51.951069) v2 -- ?+0 0x662b6c0 con 0x31e1600 -8378> 2014-05-19 18:35:46.866683 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8377> 2014-05-19 18:35:46.866631 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:51.951069) v2 ==== 47+0+0 (81849990 0 0) 0x79628c0 con 0x61b0160 -8376> 2014-05-19 18:35:46.866787 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:51.951069) v2 -- ?+0 0x5cbca80 con 0x61b0160 -8375> 2014-05-19 18:35:46.866832 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8374> 2014-05-19 18:35:46.944614 7fd0396a8700 20 OSD::command_tp worker waiting -8373> 2014-05-19 18:35:47.106300 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:52.190761) v2 ==== 47+0+0 (1685179892 0 0) 0x6710e00 con 0x30946e0 -8372> 2014-05-19 18:35:47.106320 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 49 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:52.190761) v2 ==== 47+0+0 (1685179892 0 0) 0x68aae00 con 0x3094160 -8371> 2014-05-19 18:35:47.106350 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:52.190761) v2 -- ?+0 0x79e7340 con 0x30946e0 -8370> 2014-05-19 18:35:47.106382 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8369> 2014-05-19 18:35:47.106400 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:52.190761) v2 -- ?+0 0x79628c0 con 0x3094160 -8368> 2014-05-19 18:35:47.106480 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8367> 2014-05-19 18:35:47.338955 7fd03c6ae700 20 OSD::op_tp worker waiting -8366> 2014-05-19 18:35:47.339004 7fd041eb9700 20 OSD::op_tp worker waiting -8365> 2014-05-19 18:35:47.339021 7fd03e6b2700 20 OSD::op_tp worker waiting -8364> 2014-05-19 18:35:47.339504 7fd03d6b0700 20 OSD::op_tp worker waiting -8363> 2014-05-19 18:35:47.339686 7fd0436bc700 20 OSD::op_tp worker waiting -8362> 2014-05-19 18:35:47.339744 7fd03bead700 20 OSD::op_tp worker waiting -8361> 2014-05-19 18:35:47.339784 7fd03f6b4700 20 OSD::op_tp worker waiting -8360> 2014-05-19 18:35:47.353424 7fd0446be700 20 OSD::op_tp worker waiting -8359> 2014-05-19 18:35:47.353464 7fd03feb5700 20 OSD::op_tp worker waiting -8358> 2014-05-19 18:35:47.360562 7fd040eb7700 20 OSD::op_tp worker waiting -8357> 2014-05-19 18:35:47.365678 7fd0406b6700 20 OSD::op_tp worker waiting -8356> 2014-05-19 18:35:47.387373 7fd039ea9700 20 OSD::disk_tp worker waiting -8355> 2014-05-19 18:35:47.390845 7fd03aeab700 20 OSD::op_tp worker waiting -8354> 2014-05-19 18:35:47.390871 7fd03eeb3700 20 OSD::op_tp worker waiting -8353> 2014-05-19 18:35:47.408787 7fd042ebb700 20 OSD::op_tp worker waiting -8352> 2014-05-19 18:35:47.408812 7fd03deb1700 20 OSD::op_tp worker waiting -8351> 2014-05-19 18:35:47.417272 7fd04e104700 20 KeyValueStore::op_tp worker waiting -8350> 2014-05-19 18:35:47.417275 7fd0426ba700 20 OSD::op_tp worker waiting -8349> 2014-05-19 18:35:47.417310 7fd03b6ac700 20 OSD::op_tp worker waiting -8348> 2014-05-19 18:35:47.417329 7fd03ceaf700 20 OSD::op_tp worker waiting -8347> 2014-05-19 18:35:47.417359 7fd0416b8700 20 OSD::op_tp worker waiting -8346> 2014-05-19 18:35:47.484530 7fd04f907700 5 osd.0 21 tick -8345> 2014-05-19 18:35:47.484547 7fd04f907700 7 osd.0 21 do_mon_report -8344> 2014-05-19 18:35:47.484551 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -8343> 2014-05-19 18:35:47.484555 7fd04f907700 20 osd.0 21 send_pg_stats -8342> 2014-05-19 18:35:47.484563 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8341> 2014-05-19 18:35:47.484567 7fd04f907700 10 osd.0 21 do_waiters -- start -8340> 2014-05-19 18:35:47.484569 7fd04f907700 10 osd.0 21 do_waiters -- finish -8339> 2014-05-19 18:35:47.484607 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8338> 2014-05-19 18:35:47.777354 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:47.777141) v2 ==== 47+0+0 (3681800989 0 0) 0x60f68c0 con 0x4cb86e0 -8337> 2014-05-19 18:35:47.777364 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:47.777141) v2 ==== 47+0+0 (3681800989 0 0) 0x6009880 con 0x4cb8420 -8336> 2014-05-19 18:35:47.777417 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:47.777141) v2 -- ?+0 0x6710e00 con 0x4cb86e0 -8335> 2014-05-19 18:35:47.777469 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8334> 2014-05-19 18:35:47.777486 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:47.777141) v2 -- ?+0 0x68aae00 con 0x4cb8420 -8333> 2014-05-19 18:35:47.777515 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8332> 2014-05-19 18:35:47.914348 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:50.970802) v2 ==== 47+0+0 (2972240489 0 0) 0x5bea1c0 con 0x61ae420 -8331> 2014-05-19 18:35:47.914388 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:50.970802) v2 -- ?+0 0x60f68c0 con 0x61ae420 -8330> 2014-05-19 18:35:47.914413 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8329> 2014-05-19 18:35:47.914455 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:50.970802) v2 ==== 47+0+0 (2972240489 0 0) 0x7a37c00 con 0x618fe40 -8328> 2014-05-19 18:35:47.914504 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:50.970802) v2 -- ?+0 0x6009880 con 0x618fe40 -8327> 2014-05-19 18:35:47.914541 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8326> 2014-05-19 18:35:48.063398 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:48.063208) v2 ==== 47+0+0 (749756664 0 0) 0x662b180 con 0x4cb8160 -8325> 2014-05-19 18:35:48.063424 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:48.063208) v2 ==== 47+0+0 (749756664 0 0) 0x5cbc540 con 0x4cb82c0 -8324> 2014-05-19 18:35:48.063437 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:48.063208) v2 -- ?+0 0x5bea1c0 con 0x4cb8160 -8323> 2014-05-19 18:35:48.063499 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8322> 2014-05-19 18:35:48.063541 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:48.063208) v2 -- ?+0 0x7a37c00 con 0x4cb82c0 -8321> 2014-05-19 18:35:48.063578 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8320> 2014-05-19 18:35:48.277846 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:48.277683) v2 ==== 47+0+0 (3202136492 0 0) 0x784c380 con 0x4cb86e0 -8319> 2014-05-19 18:35:48.277879 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:48.277683) v2 ==== 47+0+0 (3202136492 0 0) 0x5c041c0 con 0x4cb8420 -8318> 2014-05-19 18:35:48.277895 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:48.277683) v2 -- ?+0 0x662b180 con 0x4cb86e0 -8317> 2014-05-19 18:35:48.277928 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8316> 2014-05-19 18:35:48.277951 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:48.277683) v2 -- ?+0 0x5cbc540 con 0x4cb8420 -8315> 2014-05-19 18:35:48.277988 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8314> 2014-05-19 18:35:48.484635 7fd04f907700 5 osd.0 21 tick -8313> 2014-05-19 18:35:48.484684 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.13 < max 0.5 = yes -8312> 2014-05-19 18:35:48.484697 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -8311> 2014-05-19 18:35:48.484704 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8310> 2014-05-19 18:35:48.484700 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 191.209 < min (86400 seconds) -8309> 2014-05-19 18:35:48.484719 7fd04f907700 20 osd.0 21 sched_scrub done -8308> 2014-05-19 18:35:48.484722 7fd04f907700 10 osd.0 21 do_waiters -- start -8307> 2014-05-19 18:35:48.484724 7fd04f907700 10 osd.0 21 do_waiters -- finish -8306> 2014-05-19 18:35:48.772466 7fd043ebd700 20 OSD::op_tp worker waiting -8305> 2014-05-19 18:35:48.772481 7fd04e905700 20 KeyValueStore::op_tp worker waiting -8304> 2014-05-19 18:35:48.944691 7fd0396a8700 20 OSD::command_tp worker waiting -8303> 2014-05-19 18:35:49.339006 7fd03c6ae700 20 OSD::op_tp worker waiting -8302> 2014-05-19 18:35:49.339081 7fd041eb9700 20 OSD::op_tp worker waiting -8301> 2014-05-19 18:35:49.339111 7fd03e6b2700 20 OSD::op_tp worker waiting -8300> 2014-05-19 18:35:49.339574 7fd03d6b0700 20 OSD::op_tp worker waiting -8299> 2014-05-19 18:35:49.339754 7fd0436bc700 20 OSD::op_tp worker waiting -8298> 2014-05-19 18:35:49.339812 7fd03bead700 20 OSD::op_tp worker waiting -8297> 2014-05-19 18:35:49.339853 7fd03f6b4700 20 OSD::op_tp worker waiting -8296> 2014-05-19 18:35:49.352443 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:52.408803) v2 ==== 47+0+0 (1897831596 0 0) 0x2ee1dc0 con 0x4cb8dc0 -8295> 2014-05-19 18:35:49.352471 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:52.408803) v2 ==== 47+0+0 (1897831596 0 0) 0x5497340 con 0x4cb9340 -8294> 2014-05-19 18:35:49.352493 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:52.408803) v2 -- ?+0 0x784c380 con 0x4cb8dc0 -8293> 2014-05-19 18:35:49.352568 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8292> 2014-05-19 18:35:49.352646 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:52.408803) v2 -- ?+0 0x5c041c0 con 0x4cb9340 -8291> 2014-05-19 18:35:49.352676 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8290> 2014-05-19 18:35:49.353504 7fd0446be700 20 OSD::op_tp worker waiting -8289> 2014-05-19 18:35:49.353567 7fd03feb5700 20 OSD::op_tp worker waiting -8288> 2014-05-19 18:35:49.360649 7fd040eb7700 20 OSD::op_tp worker waiting -8287> 2014-05-19 18:35:49.365775 7fd0406b6700 20 OSD::op_tp worker waiting -8286> 2014-05-19 18:35:49.387459 7fd039ea9700 20 OSD::disk_tp worker waiting -8285> 2014-05-19 18:35:49.390947 7fd03aeab700 20 OSD::op_tp worker waiting -8284> 2014-05-19 18:35:49.390971 7fd03eeb3700 20 OSD::op_tp worker waiting -8283> 2014-05-19 18:35:49.408888 7fd042ebb700 20 OSD::op_tp worker waiting -8282> 2014-05-19 18:35:49.408911 7fd03deb1700 20 OSD::op_tp worker waiting -8281> 2014-05-19 18:35:49.417378 7fd04e104700 20 KeyValueStore::op_tp worker waiting -8280> 2014-05-19 18:35:49.417381 7fd0426ba700 20 OSD::op_tp worker waiting -8279> 2014-05-19 18:35:49.417465 7fd03b6ac700 20 OSD::op_tp worker waiting -8278> 2014-05-19 18:35:49.417487 7fd03ceaf700 20 OSD::op_tp worker waiting -8277> 2014-05-19 18:35:49.417543 7fd0416b8700 20 OSD::op_tp worker waiting -8276> 2014-05-19 18:35:49.484796 7fd04f907700 5 osd.0 21 tick -8275> 2014-05-19 18:35:49.484817 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8274> 2014-05-19 18:35:49.484822 7fd04f907700 10 osd.0 21 do_waiters -- start -8273> 2014-05-19 18:35:49.484824 7fd04f907700 10 osd.0 21 do_waiters -- finish -8272> 2014-05-19 18:35:49.484865 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8271> 2014-05-19 18:35:49.648462 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -8270> 2014-05-19 18:35:49.648480 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -8269> 2014-05-19 18:35:49.648488 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x612ca80 con 0x3622580 -8268> 2014-05-19 18:35:49.648514 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x6008540 con 0x3622840 -8267> 2014-05-19 18:35:49.648553 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x55c3500 con 0x3622b00 -8266> 2014-05-19 18:35:49.648568 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x6b71340 con 0x3622dc0 -8265> 2014-05-19 18:35:49.648635 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x6711500 con 0x30594a0 -8264> 2014-05-19 18:35:49.648669 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x64d3180 con 0x3622f20 -8263> 2014-05-19 18:35:49.648697 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x6efe540 con 0x5e7ef20 -8262> 2014-05-19 18:35:49.648723 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x7747880 con 0x5e7f1e0 -8261> 2014-05-19 18:35:49.648750 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x6fe61c0 con 0x5e7e2c0 -8260> 2014-05-19 18:35:49.648776 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x55c36c0 con 0x5d7bb80 -8259> 2014-05-19 18:35:49.648793 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x73c2540 con 0x5e7ec60 -8258> 2014-05-19 18:35:49.648807 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x72e4540 con 0x531b760 -8257> 2014-05-19 18:35:49.648830 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x8389880 con 0x53bba20 -8256> 2014-05-19 18:35:49.648855 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x6378a80 con 0x51cac60 -8255> 2014-05-19 18:35:49.648870 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x77476c0 con 0x3622580 -8254> 2014-05-19 18:35:49.648896 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x692d340 con 0x5bd66e0 -8253> 2014-05-19 18:35:49.648921 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:49.648486) v2 -- ?+0 0x6c4c000 con 0x5bd6840 -8252> 2014-05-19 18:35:49.648951 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8251> 2014-05-19 18:35:49.649003 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x5f7b340 con 0x3622b00 -8250> 2014-05-19 18:35:49.649036 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8249> 2014-05-19 18:35:49.649077 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x7970540 con 0x3622dc0 -8248> 2014-05-19 18:35:49.649107 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8247> 2014-05-19 18:35:49.649123 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x5f32380 con 0x3622840 -8246> 2014-05-19 18:35:49.649154 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8245> 2014-05-19 18:35:49.649167 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x612c8c0 con 0x5e7e2c0 -8244> 2014-05-19 18:35:49.649195 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8243> 2014-05-19 18:35:49.649209 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x6c4d6c0 con 0x30594a0 -8242> 2014-05-19 18:35:49.649307 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8241> 2014-05-19 18:35:49.649332 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x6009dc0 con 0x5d7bb80 -8240> 2014-05-19 18:35:49.649362 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8239> 2014-05-19 18:35:49.649378 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x6542fc0 con 0x5e7ec60 -8238> 2014-05-19 18:35:49.649408 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8237> 2014-05-19 18:35:49.649422 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x788a000 con 0x531b760 -8236> 2014-05-19 18:35:49.649449 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8235> 2014-05-19 18:35:49.649464 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x62e0540 con 0x53bba20 -8234> 2014-05-19 18:35:49.649494 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8233> 2014-05-19 18:35:49.649512 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x70a81c0 con 0x51cac60 -8232> 2014-05-19 18:35:49.649543 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8231> 2014-05-19 18:35:49.649557 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x60a1a40 con 0x5bd66e0 -8230> 2014-05-19 18:35:49.649588 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8229> 2014-05-19 18:35:49.649603 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 56 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x71236c0 con 0x5bd6840 -8228> 2014-05-19 18:35:49.649634 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8227> 2014-05-19 18:35:49.649648 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x59bd500 con 0x3622f20 -8226> 2014-05-19 18:35:49.649678 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8225> 2014-05-19 18:35:49.649690 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x6504e00 con 0x5e7ef20 -8224> 2014-05-19 18:35:49.649720 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8223> 2014-05-19 18:35:49.649733 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.648486) v2 ==== 47+0+0 (2025444054 0 0) 0x7930540 con 0x5e7f1e0 -8222> 2014-05-19 18:35:49.649763 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8221> 2014-05-19 18:35:49.906621 7fd0456c0700 10 monclient: tick -8220> 2014-05-19 18:35:49.906634 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:35:19.906633) -8219> 2014-05-19 18:35:49.906651 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:35:49.906651; renew after: 2014-05-19 18:35:39.994233) -- yes -8218> 2014-05-19 18:35:49.906662 7fd0456c0700 10 monclient: renew_subs -8217> 2014-05-19 18:35:49.906667 7fd0456c0700 10 monclient: _send_mon_message to mon.a at 172.16.96.201:6789/0 -8216> 2014-05-19 18:35:49.906683 7fd0456c0700 1 -- 172.16.96.201:6800/3074 --> 172.16.96.201:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0}) v2 -- ?+0 0x60a1340 con 0x2e3edc0 -8215> 2014-05-19 18:35:49.907086 7fd04aecb700 1 -- 172.16.96.201:6800/3074 <== mon.0 172.16.96.201:6789/0 44 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4198368508 0 0) 0x7890540 con 0x2e3edc0 -8214> 2014-05-19 18:35:49.907114 7fd04aecb700 10 monclient: handle_subscribe_ack sent 2014-05-19 18:35:49.906664 renew after 2014-05-19 18:38:19.906664 -8213> 2014-05-19 18:35:49.978463 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:49.978284) v2 ==== 47+0+0 (4265269805 0 0) 0x784d6c0 con 0x4cb86e0 -8212> 2014-05-19 18:35:49.978504 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:49.978284) v2 ==== 47+0+0 (4265269805 0 0) 0x5c04000 con 0x4cb8420 -8211> 2014-05-19 18:35:49.978514 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.978284) v2 -- ?+0 0x2ee1dc0 con 0x4cb86e0 -8210> 2014-05-19 18:35:49.978566 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8209> 2014-05-19 18:35:49.978610 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:49.978284) v2 -- ?+0 0x5497340 con 0x4cb8420 -8208> 2014-05-19 18:35:49.978726 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8207> 2014-05-19 18:35:50.484891 7fd04f907700 5 osd.0 21 tick -8206> 2014-05-19 18:35:50.484939 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.13 < max 0.5 = yes -8205> 2014-05-19 18:35:50.484953 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -8204> 2014-05-19 18:35:50.484948 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8203> 2014-05-19 18:35:50.484956 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 193.209 < min (86400 seconds) -8202> 2014-05-19 18:35:50.484975 7fd04f907700 20 osd.0 21 sched_scrub done -8201> 2014-05-19 18:35:50.484977 7fd04f907700 10 osd.0 21 do_waiters -- start -8200> 2014-05-19 18:35:50.484979 7fd04f907700 10 osd.0 21 do_waiters -- finish -8199> 2014-05-19 18:35:50.772553 7fd04e905700 20 KeyValueStore::op_tp worker waiting -8198> 2014-05-19 18:35:50.772557 7fd043ebd700 20 OSD::op_tp worker waiting -8197> 2014-05-19 18:35:50.881564 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:55.966149) v2 ==== 47+0+0 (737204447 0 0) 0x60e2000 con 0x5f5b760 -8196> 2014-05-19 18:35:50.881568 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:55.966149) v2 ==== 47+0+0 (737204447 0 0) 0x60e01c0 con 0x5b21600 -8195> 2014-05-19 18:35:50.881622 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:55.966149) v2 -- ?+0 0x5c04000 con 0x5f5b760 -8194> 2014-05-19 18:35:50.881657 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8193> 2014-05-19 18:35:50.881680 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:55.966149) v2 -- ?+0 0x784d6c0 con 0x5b21600 -8192> 2014-05-19 18:35:50.881751 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8191> 2014-05-19 18:35:50.944782 7fd0396a8700 20 OSD::command_tp worker waiting -8190> 2014-05-19 18:35:51.052962 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:54.109345) v2 ==== 47+0+0 (732308038 0 0) 0x2ee16c0 con 0x4cb8dc0 -8189> 2014-05-19 18:35:51.052981 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:54.109345) v2 ==== 47+0+0 (732308038 0 0) 0x5496c40 con 0x4cb9340 -8188> 2014-05-19 18:35:51.053002 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:54.109345) v2 -- ?+0 0x60e01c0 con 0x4cb8dc0 -8187> 2014-05-19 18:35:51.053049 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8186> 2014-05-19 18:35:51.053091 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:54.109345) v2 -- ?+0 0x60e2000 con 0x4cb9340 -8185> 2014-05-19 18:35:51.053139 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8184> 2014-05-19 18:35:51.283717 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:54.340136) v2 ==== 47+0+0 (2497929879 0 0) 0x70cc1c0 con 0x61b0420 -8183> 2014-05-19 18:35:51.283730 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:54.340136) v2 ==== 47+0+0 (2497929879 0 0) 0x5666c40 con 0x61ae160 -8182> 2014-05-19 18:35:51.283773 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:54.340136) v2 -- ?+0 0x5496c40 con 0x61b0420 -8181> 2014-05-19 18:35:51.283812 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8180> 2014-05-19 18:35:51.283857 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:54.340136) v2 -- ?+0 0x2ee16c0 con 0x61ae160 -8179> 2014-05-19 18:35:51.283893 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8178> 2014-05-19 18:35:51.339100 7fd03c6ae700 20 OSD::op_tp worker waiting -8177> 2014-05-19 18:35:51.339175 7fd041eb9700 20 OSD::op_tp worker waiting -8176> 2014-05-19 18:35:51.339222 7fd03e6b2700 20 OSD::op_tp worker waiting -8175> 2014-05-19 18:35:51.339656 7fd03d6b0700 20 OSD::op_tp worker waiting -8174> 2014-05-19 18:35:51.339835 7fd0436bc700 20 OSD::op_tp worker waiting -8173> 2014-05-19 18:35:51.339884 7fd03bead700 20 OSD::op_tp worker waiting -8172> 2014-05-19 18:35:51.339936 7fd03f6b4700 20 OSD::op_tp worker waiting -8171> 2014-05-19 18:35:51.353601 7fd0446be700 20 OSD::op_tp worker waiting -8170> 2014-05-19 18:35:51.353653 7fd03feb5700 20 OSD::op_tp worker waiting -8169> 2014-05-19 18:35:51.360736 7fd040eb7700 20 OSD::op_tp worker waiting -8168> 2014-05-19 18:35:51.365872 7fd0406b6700 20 OSD::op_tp worker waiting -8167> 2014-05-19 18:35:51.387540 7fd039ea9700 20 OSD::disk_tp worker waiting -8166> 2014-05-19 18:35:51.391047 7fd03aeab700 20 OSD::op_tp worker waiting -8165> 2014-05-19 18:35:51.391075 7fd03eeb3700 20 OSD::op_tp worker waiting -8164> 2014-05-19 18:35:51.408989 7fd042ebb700 20 OSD::op_tp worker waiting -8163> 2014-05-19 18:35:51.409015 7fd03deb1700 20 OSD::op_tp worker waiting -8162> 2014-05-19 18:35:51.414937 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:54.471327) v2 ==== 47+0+0 (19528684 0 0) 0x7a37880 con 0x618fe40 -8161> 2014-05-19 18:35:51.414953 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:54.471327) v2 ==== 47+0+0 (19528684 0 0) 0x5beb340 con 0x61ae420 -8160> 2014-05-19 18:35:51.414987 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:54.471327) v2 -- ?+0 0x70cc1c0 con 0x618fe40 -8159> 2014-05-19 18:35:51.415021 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8158> 2014-05-19 18:35:51.415047 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:54.471327) v2 -- ?+0 0x5666c40 con 0x61ae420 -8157> 2014-05-19 18:35:51.415091 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8156> 2014-05-19 18:35:51.417496 7fd04e104700 20 KeyValueStore::op_tp worker waiting -8155> 2014-05-19 18:35:51.417498 7fd0426ba700 20 OSD::op_tp worker waiting -8154> 2014-05-19 18:35:51.417543 7fd03b6ac700 20 OSD::op_tp worker waiting -8153> 2014-05-19 18:35:51.417570 7fd03ceaf700 20 OSD::op_tp worker waiting -8152> 2014-05-19 18:35:51.417627 7fd0416b8700 20 OSD::op_tp worker waiting -8151> 2014-05-19 18:35:51.485044 7fd04f907700 5 osd.0 21 tick -8150> 2014-05-19 18:35:51.485065 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8149> 2014-05-19 18:35:51.485071 7fd04f907700 10 osd.0 21 do_waiters -- start -8148> 2014-05-19 18:35:51.485074 7fd04f907700 10 osd.0 21 do_waiters -- finish -8147> 2014-05-19 18:35:51.485102 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8146> 2014-05-19 18:35:51.563925 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:51.563760) v2 ==== 47+0+0 (3487186575 0 0) 0x662b500 con 0x4cb8160 -8145> 2014-05-19 18:35:51.563969 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:51.563760) v2 -- ?+0 0x5beb340 con 0x4cb8160 -8144> 2014-05-19 18:35:51.563993 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8143> 2014-05-19 18:35:51.563989 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:51.563760) v2 ==== 47+0+0 (3487186575 0 0) 0x5cbcfc0 con 0x4cb82c0 -8142> 2014-05-19 18:35:51.564030 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:51.563760) v2 -- ?+0 0x7a37880 con 0x4cb82c0 -8141> 2014-05-19 18:35:51.564115 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8140> 2014-05-19 18:35:51.806625 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:56.891231) v2 ==== 47+0+0 (3721201833 0 0) 0x68aaa80 con 0x3094160 -8139> 2014-05-19 18:35:51.806639 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 50 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:56.891231) v2 ==== 47+0+0 (3721201833 0 0) 0x67101c0 con 0x30946e0 -8138> 2014-05-19 18:35:51.806674 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:56.891231) v2 -- ?+0 0x5cbcfc0 con 0x3094160 -8137> 2014-05-19 18:35:51.806719 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8136> 2014-05-19 18:35:51.806742 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:56.891231) v2 -- ?+0 0x662b500 con 0x30946e0 -8135> 2014-05-19 18:35:51.806788 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8134> 2014-05-19 18:35:52.166934 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:57.251555) v2 ==== 47+0+0 (4069551894 0 0) 0x649e000 con 0x31e1600 -8133> 2014-05-19 18:35:52.166941 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:57.251555) v2 ==== 47+0+0 (4069551894 0 0) 0x5c04e00 con 0x61b0160 -8132> 2014-05-19 18:35:52.166986 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.251555) v2 -- ?+0 0x67101c0 con 0x31e1600 -8131> 2014-05-19 18:35:52.167019 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8130> 2014-05-19 18:35:52.167049 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.251555) v2 -- ?+0 0x68aaa80 con 0x61b0160 -8129> 2014-05-19 18:35:52.167099 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8128> 2014-05-19 18:35:52.485149 7fd04f907700 5 osd.0 21 tick -8127> 2014-05-19 18:35:52.485165 7fd04f907700 7 osd.0 21 do_mon_report -8126> 2014-05-19 18:35:52.485169 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -8125> 2014-05-19 18:35:52.485173 7fd04f907700 20 osd.0 21 send_pg_stats -8124> 2014-05-19 18:35:52.485181 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8123> 2014-05-19 18:35:52.485184 7fd04f907700 10 osd.0 21 do_waiters -- start -8122> 2014-05-19 18:35:52.485187 7fd04f907700 10 osd.0 21 do_waiters -- finish -8121> 2014-05-19 18:35:52.485200 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8120> 2014-05-19 18:35:52.581940 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:57.666591) v2 ==== 47+0+0 (4142767858 0 0) 0x649fa40 con 0x5b21600 -8119> 2014-05-19 18:35:52.581950 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:57.666591) v2 ==== 47+0+0 (4142767858 0 0) 0x5c46fc0 con 0x5f5b760 -8118> 2014-05-19 18:35:52.581986 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.666591) v2 -- ?+0 0x649e000 con 0x5b21600 -8117> 2014-05-19 18:35:52.582073 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8116> 2014-05-19 18:35:52.582115 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.666591) v2 -- ?+0 0x5c04e00 con 0x5f5b760 -8115> 2014-05-19 18:35:52.582160 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8114> 2014-05-19 18:35:52.772667 7fd04e905700 20 KeyValueStore::op_tp worker waiting -8113> 2014-05-19 18:35:52.772671 7fd043ebd700 20 OSD::op_tp worker waiting -8112> 2014-05-19 18:35:52.944867 7fd0396a8700 20 OSD::command_tp worker waiting -8111> 2014-05-19 18:35:52.984384 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:56.040799) v2 ==== 47+0+0 (1855611850 0 0) 0x5667a40 con 0x61ae160 -8110> 2014-05-19 18:35:52.984388 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:56.040799) v2 ==== 47+0+0 (1855611850 0 0) 0x70cda40 con 0x61b0420 -8109> 2014-05-19 18:35:52.984437 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:56.040799) v2 -- ?+0 0x649fa40 con 0x61ae160 -8108> 2014-05-19 18:35:52.984477 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8107> 2014-05-19 18:35:52.984530 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:56.040799) v2 -- ?+0 0x5c46fc0 con 0x61b0420 -8106> 2014-05-19 18:35:52.984612 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8105> 2014-05-19 18:35:53.149040 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -8104> 2014-05-19 18:35:53.149059 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -8103> 2014-05-19 18:35:53.149067 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x55181c0 con 0x3622580 -8102> 2014-05-19 18:35:53.149093 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x3510380 con 0x3622840 -8101> 2014-05-19 18:35:53.149131 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x7a5e8c0 con 0x3622b00 -8100> 2014-05-19 18:35:53.149146 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x6c4a700 con 0x3622dc0 -8099> 2014-05-19 18:35:53.149160 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x6543880 con 0x30594a0 -8098> 2014-05-19 18:35:53.149175 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x8388540 con 0x3622f20 -8097> 2014-05-19 18:35:53.149191 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x711e8c0 con 0x5e7ef20 -8096> 2014-05-19 18:35:53.149218 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x77461c0 con 0x5e7f1e0 -8095> 2014-05-19 18:35:53.149248 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x798e380 con 0x5e7e2c0 -8094> 2014-05-19 18:35:53.149272 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x6fe7880 con 0x5d7bb80 -8093> 2014-05-19 18:35:53.149297 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x60e0700 con 0x5e7ec60 -8092> 2014-05-19 18:35:53.149324 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x72e5dc0 con 0x531b760 -8091> 2014-05-19 18:35:53.149351 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x79e68c0 con 0x53bba20 -8090> 2014-05-19 18:35:53.149398 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x66f88c0 con 0x51cac60 -8089> 2014-05-19 18:35:53.149428 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x73c2380 con 0x5bd66e0 -8088> 2014-05-19 18:35:53.149462 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:53.149065) v2 -- ?+0 0x6c4cc40 con 0x5bd6840 -8087> 2014-05-19 18:35:53.149516 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x7971500 con 0x3622dc0 -8086> 2014-05-19 18:35:53.149570 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8085> 2014-05-19 18:35:53.149605 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x7746c40 con 0x3622580 -8084> 2014-05-19 18:35:53.149630 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8083> 2014-05-19 18:35:53.149655 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x6b70540 con 0x3622840 -8082> 2014-05-19 18:35:53.149674 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -8081> 2014-05-19 18:35:53.149682 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x5f7a8c0 con 0x3622b00 -8080> 2014-05-19 18:35:53.149697 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8079> 2014-05-19 18:35:53.149705 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x6504700 con 0x5e7ef20 -8078> 2014-05-19 18:35:53.149719 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8077> 2014-05-19 18:35:53.149726 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x612ce00 con 0x5e7e2c0 -8076> 2014-05-19 18:35:53.149742 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8075> 2014-05-19 18:35:53.149747 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x6009180 con 0x5d7bb80 -8074> 2014-05-19 18:35:53.149762 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -8073> 2014-05-19 18:35:53.149767 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x6c4d180 con 0x30594a0 -8072> 2014-05-19 18:35:53.149854 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8071> 2014-05-19 18:35:53.149900 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x3438700 con 0x53bba20 -8070> 2014-05-19 18:35:53.149929 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8069> 2014-05-19 18:35:53.149940 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x77e8000 con 0x51cac60 -8068> 2014-05-19 18:35:53.149960 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8067> 2014-05-19 18:35:53.149973 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x60a0700 con 0x5bd66e0 -8066> 2014-05-19 18:35:53.149992 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8065> 2014-05-19 18:35:53.149998 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 57 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x7122a80 con 0x5bd6840 -8064> 2014-05-19 18:35:53.150029 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -8063> 2014-05-19 18:35:53.150042 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x59bddc0 con 0x3622f20 -8062> 2014-05-19 18:35:53.150058 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -8061> 2014-05-19 18:35:53.150066 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x7931a40 con 0x5e7f1e0 -8060> 2014-05-19 18:35:53.150080 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -8059> 2014-05-19 18:35:53.150087 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x6543180 con 0x5e7ec60 -8058> 2014-05-19 18:35:53.150104 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8057> 2014-05-19 18:35:53.150115 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.149065) v2 ==== 47+0+0 (3743895484 0 0) 0x788a540 con 0x531b760 -8056> 2014-05-19 18:35:53.150129 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8055> 2014-05-19 18:35:53.339175 7fd03c6ae700 20 OSD::op_tp worker waiting -8054> 2014-05-19 18:35:53.339264 7fd041eb9700 20 OSD::op_tp worker waiting -8053> 2014-05-19 18:35:53.339321 7fd03e6b2700 20 OSD::op_tp worker waiting -8052> 2014-05-19 18:35:53.339726 7fd03d6b0700 20 OSD::op_tp worker waiting -8051> 2014-05-19 18:35:53.339915 7fd0436bc700 20 OSD::op_tp worker waiting -8050> 2014-05-19 18:35:53.339983 7fd03bead700 20 OSD::op_tp worker waiting -8049> 2014-05-19 18:35:53.340039 7fd03f6b4700 20 OSD::op_tp worker waiting -8048> 2014-05-19 18:35:53.353694 7fd0446be700 20 OSD::op_tp worker waiting -8047> 2014-05-19 18:35:53.353731 7fd03feb5700 20 OSD::op_tp worker waiting -8046> 2014-05-19 18:35:53.360832 7fd040eb7700 20 OSD::op_tp worker waiting -8045> 2014-05-19 18:35:53.365970 7fd0406b6700 20 OSD::op_tp worker waiting -8044> 2014-05-19 18:35:53.387626 7fd039ea9700 20 OSD::disk_tp worker waiting -8043> 2014-05-19 18:35:53.391148 7fd03aeab700 20 OSD::op_tp worker waiting -8042> 2014-05-19 18:35:53.391173 7fd03eeb3700 20 OSD::op_tp worker waiting -8041> 2014-05-19 18:35:53.409072 7fd042ebb700 20 OSD::op_tp worker waiting -8040> 2014-05-19 18:35:53.409096 7fd03deb1700 20 OSD::op_tp worker waiting -8039> 2014-05-19 18:35:53.417602 7fd04e104700 20 KeyValueStore::op_tp worker waiting -8038> 2014-05-19 18:35:53.417605 7fd0426ba700 20 OSD::op_tp worker waiting -8037> 2014-05-19 18:35:53.417641 7fd03b6ac700 20 OSD::op_tp worker waiting -8036> 2014-05-19 18:35:53.417663 7fd03ceaf700 20 OSD::op_tp worker waiting -8035> 2014-05-19 18:35:53.417704 7fd0416b8700 20 OSD::op_tp worker waiting -8034> 2014-05-19 18:35:53.479012 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:53.478840) v2 ==== 47+0+0 (634340420 0 0) 0x75b08c0 con 0x4cb86e0 -8033> 2014-05-19 18:35:53.479059 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.478840) v2 -- ?+0 0x5667a40 con 0x4cb86e0 -8032> 2014-05-19 18:35:53.479093 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8031> 2014-05-19 18:35:53.479082 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:53.478840) v2 ==== 47+0+0 (634340420 0 0) 0x5c05340 con 0x4cb8420 -8030> 2014-05-19 18:35:53.479116 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:53.478840) v2 -- ?+0 0x70cda40 con 0x4cb8420 -8029> 2014-05-19 18:35:53.479202 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -8028> 2014-05-19 18:35:53.485259 7fd04f907700 5 osd.0 21 tick -8027> 2014-05-19 18:35:53.485309 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.12 < max 0.5 = yes -8026> 2014-05-19 18:35:53.485323 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -8025> 2014-05-19 18:35:53.485328 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8024> 2014-05-19 18:35:53.485326 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 196.209 < min (86400 seconds) -8023> 2014-05-19 18:35:53.485344 7fd04f907700 20 osd.0 21 sched_scrub done -8022> 2014-05-19 18:35:53.485347 7fd04f907700 10 osd.0 21 do_waiters -- start -8021> 2014-05-19 18:35:53.485349 7fd04f907700 10 osd.0 21 do_waiters -- finish -8020> 2014-05-19 18:35:53.506962 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:58.591691) v2 ==== 47+0+0 (1821971251 0 0) 0x649e540 con 0x30946e0 -8019> 2014-05-19 18:35:53.507007 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.591691) v2 -- ?+0 0x75b08c0 con 0x30946e0 -8018> 2014-05-19 18:35:53.507014 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 51 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:58.591691) v2 ==== 47+0+0 (1821971251 0 0) 0x6e188c0 con 0x3094160 -8017> 2014-05-19 18:35:53.507085 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8016> 2014-05-19 18:35:53.507113 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.591691) v2 -- ?+0 0x5c05340 con 0x3094160 -8015> 2014-05-19 18:35:53.507156 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -8014> 2014-05-19 18:35:53.867401 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:58.952113) v2 ==== 47+0+0 (530389307 0 0) 0x5c05a40 con 0x61b0160 -8013> 2014-05-19 18:35:53.867414 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:58.952113) v2 ==== 47+0+0 (530389307 0 0) 0x7947dc0 con 0x31e1600 -8012> 2014-05-19 18:35:53.867450 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.952113) v2 -- ?+0 0x6e188c0 con 0x61b0160 -8011> 2014-05-19 18:35:53.867486 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8010> 2014-05-19 18:35:53.867527 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.952113) v2 -- ?+0 0x649e540 con 0x31e1600 -8009> 2014-05-19 18:35:53.867569 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -8008> 2014-05-19 18:35:54.485433 7fd04f907700 5 osd.0 21 tick -8007> 2014-05-19 18:35:54.485453 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -8006> 2014-05-19 18:35:54.485458 7fd04f907700 10 osd.0 21 do_waiters -- start -8005> 2014-05-19 18:35:54.485460 7fd04f907700 10 osd.0 21 do_waiters -- finish -8004> 2014-05-19 18:35:54.485501 7fd03a6aa700 20 OSD::recovery_tp worker waiting -8003> 2014-05-19 18:35:54.553429 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:57.609835) v2 ==== 47+0+0 (2949987969 0 0) 0x73c2e00 con 0x4cb9340 -8002> 2014-05-19 18:35:54.553430 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:57.609835) v2 ==== 47+0+0 (2949987969 0 0) 0x2ee0fc0 con 0x4cb8dc0 -8001> 2014-05-19 18:35:54.553481 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.609835) v2 -- ?+0 0x5c05a40 con 0x4cb9340 -8000> 2014-05-19 18:35:54.553564 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7999> 2014-05-19 18:35:54.553591 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.609835) v2 -- ?+0 0x7947dc0 con 0x4cb8dc0 -7998> 2014-05-19 18:35:54.553645 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7997> 2014-05-19 18:35:54.772753 7fd043ebd700 20 OSD::op_tp worker waiting -7996> 2014-05-19 18:35:54.772755 7fd04e905700 20 KeyValueStore::op_tp worker waiting -7995> 2014-05-19 18:35:54.882295 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:59.967026) v2 ==== 47+0+0 (2367295232 0 0) 0x5666540 con 0x5f5b760 -7994> 2014-05-19 18:35:54.882298 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:59.967026) v2 ==== 47+0+0 (2367295232 0 0) 0x79e6a80 con 0x5b21600 -7993> 2014-05-19 18:35:54.882335 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:59.967026) v2 -- ?+0 0x73c2e00 con 0x5f5b760 -7992> 2014-05-19 18:35:54.882369 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7991> 2014-05-19 18:35:54.882426 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:59.967026) v2 -- ?+0 0x2ee0fc0 con 0x5b21600 -7990> 2014-05-19 18:35:54.882469 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7989> 2014-05-19 18:35:54.915425 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:57.971862) v2 ==== 47+0+0 (2857887526 0 0) 0x5beaa80 con 0x61ae420 -7988> 2014-05-19 18:35:54.915440 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:57.971862) v2 ==== 47+0+0 (2857887526 0 0) 0x7a36700 con 0x618fe40 -7987> 2014-05-19 18:35:54.915471 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.971862) v2 -- ?+0 0x79e6a80 con 0x61ae420 -7986> 2014-05-19 18:35:54.915495 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7985> 2014-05-19 18:35:54.915518 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.971862) v2 -- ?+0 0x5666540 con 0x618fe40 -7984> 2014-05-19 18:35:54.915557 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7983> 2014-05-19 18:35:54.944960 7fd0396a8700 20 OSD::command_tp worker waiting -7982> 2014-05-19 18:35:55.064567 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:55.064404) v2 ==== 47+0+0 (3163757759 0 0) 0x5dfee00 con 0x4cb8160 -7981> 2014-05-19 18:35:55.064654 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:55.064404) v2 -- ?+0 0x5beaa80 con 0x4cb8160 -7980> 2014-05-19 18:35:55.064657 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:55.064404) v2 ==== 47+0+0 (3163757759 0 0) 0x5cbc000 con 0x4cb82c0 -7979> 2014-05-19 18:35:55.064693 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7978> 2014-05-19 18:35:55.064718 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:55.064404) v2 -- ?+0 0x7a36700 con 0x4cb82c0 -7977> 2014-05-19 18:35:55.064818 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7976> 2014-05-19 18:35:55.339256 7fd03c6ae700 20 OSD::op_tp worker waiting -7975> 2014-05-19 18:35:55.339342 7fd041eb9700 20 OSD::op_tp worker waiting -7974> 2014-05-19 18:35:55.339405 7fd03e6b2700 20 OSD::op_tp worker waiting -7973> 2014-05-19 18:35:55.339795 7fd03d6b0700 20 OSD::op_tp worker waiting -7972> 2014-05-19 18:35:55.339996 7fd0436bc700 20 OSD::op_tp worker waiting -7971> 2014-05-19 18:35:55.340071 7fd03bead700 20 OSD::op_tp worker waiting -7970> 2014-05-19 18:35:55.340104 7fd03f6b4700 20 OSD::op_tp worker waiting -7969> 2014-05-19 18:35:55.353790 7fd0446be700 20 OSD::op_tp worker waiting -7968> 2014-05-19 18:35:55.353807 7fd03feb5700 20 OSD::op_tp worker waiting -7967> 2014-05-19 18:35:55.360924 7fd040eb7700 20 OSD::op_tp worker waiting -7966> 2014-05-19 18:35:55.366067 7fd0406b6700 20 OSD::op_tp worker waiting -7965> 2014-05-19 18:35:55.382790 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:00.467539) v2 ==== 47+0+0 (984243454 0 0) 0x72e4c40 con 0x5f5b760 -7964> 2014-05-19 18:35:55.382846 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:00.467539) v2 -- ?+0 0x5cbc000 con 0x5f5b760 -7963> 2014-05-19 18:35:55.382794 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:00.467539) v2 ==== 47+0+0 (984243454 0 0) 0x6112e00 con 0x5b21600 -7962> 2014-05-19 18:35:55.382921 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7961> 2014-05-19 18:35:55.382958 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:00.467539) v2 -- ?+0 0x5dfee00 con 0x5b21600 -7960> 2014-05-19 18:35:55.383000 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7959> 2014-05-19 18:35:55.387723 7fd039ea9700 20 OSD::disk_tp worker waiting -7958> 2014-05-19 18:35:55.391252 7fd03aeab700 20 OSD::op_tp worker waiting -7957> 2014-05-19 18:35:55.391275 7fd03eeb3700 20 OSD::op_tp worker waiting -7956> 2014-05-19 18:35:55.409172 7fd042ebb700 20 OSD::op_tp worker waiting -7955> 2014-05-19 18:35:55.409196 7fd03deb1700 20 OSD::op_tp worker waiting -7954> 2014-05-19 18:35:55.417710 7fd04e104700 20 KeyValueStore::op_tp worker waiting -7953> 2014-05-19 18:35:55.417713 7fd0426ba700 20 OSD::op_tp worker waiting -7952> 2014-05-19 18:35:55.417749 7fd03b6ac700 20 OSD::op_tp worker waiting -7951> 2014-05-19 18:35:55.417768 7fd03ceaf700 20 OSD::op_tp worker waiting -7950> 2014-05-19 18:35:55.417783 7fd0416b8700 20 OSD::op_tp worker waiting -7949> 2014-05-19 18:35:55.485546 7fd04f907700 5 osd.0 21 tick -7948> 2014-05-19 18:35:55.485567 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7947> 2014-05-19 18:35:55.485572 7fd04f907700 10 osd.0 21 do_waiters -- start -7946> 2014-05-19 18:35:55.485574 7fd04f907700 10 osd.0 21 do_waiters -- finish -7945> 2014-05-19 18:35:55.485615 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7944> 2014-05-19 18:35:55.807377 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:00.892173) v2 ==== 47+0+0 (3465833361 0 0) 0x5dff340 con 0x30946e0 -7943> 2014-05-19 18:35:55.807402 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 52 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:00.892173) v2 ==== 47+0+0 (3465833361 0 0) 0x5dea8c0 con 0x3094160 -7942> 2014-05-19 18:35:55.807428 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:00.892173) v2 -- ?+0 0x6112e00 con 0x30946e0 -7941> 2014-05-19 18:35:55.807462 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7940> 2014-05-19 18:35:55.807486 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:00.892173) v2 -- ?+0 0x72e4c40 con 0x3094160 -7939> 2014-05-19 18:35:55.807554 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7938> 2014-05-19 18:35:56.307845 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:01.392654) v2 ==== 47+0+0 (3252211687 0 0) 0x5dff180 con 0x30946e0 -7937> 2014-05-19 18:35:56.307865 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 53 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:01.392654) v2 ==== 47+0+0 (3252211687 0 0) 0x55c8700 con 0x3094160 -7936> 2014-05-19 18:35:56.307894 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:01.392654) v2 -- ?+0 0x5dff340 con 0x30946e0 -7935> 2014-05-19 18:35:56.307937 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7934> 2014-05-19 18:35:56.307956 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:01.392654) v2 -- ?+0 0x5dea8c0 con 0x3094160 -7933> 2014-05-19 18:35:56.307994 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7932> 2014-05-19 18:35:56.483183 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:01.568002) v2 ==== 47+0+0 (2017179578 0 0) 0x6113180 con 0x5b21600 -7931> 2014-05-19 18:35:56.483199 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:01.568002) v2 ==== 47+0+0 (2017179578 0 0) 0x55f5c00 con 0x5f5b760 -7930> 2014-05-19 18:35:56.483231 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:01.568002) v2 -- ?+0 0x5dff180 con 0x5b21600 -7929> 2014-05-19 18:35:56.483266 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7928> 2014-05-19 18:35:56.483296 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:01.568002) v2 -- ?+0 0x55c8700 con 0x5f5b760 -7927> 2014-05-19 18:35:56.483339 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7926> 2014-05-19 18:35:56.485005 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:59.541493) v2 ==== 47+0+0 (3643544018 0 0) 0x798e1c0 con 0x61ae160 -7925> 2014-05-19 18:35:56.485060 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:59.541493) v2 -- ?+0 0x6113180 con 0x61ae160 -7924> 2014-05-19 18:35:56.485097 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7923> 2014-05-19 18:35:56.485080 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:59.541493) v2 ==== 47+0+0 (3643544018 0 0) 0x70cd500 con 0x61b0420 -7922> 2014-05-19 18:35:56.485158 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:59.541493) v2 -- ?+0 0x55f5c00 con 0x61b0420 -7921> 2014-05-19 18:35:56.485195 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7920> 2014-05-19 18:35:56.485637 7fd04f907700 5 osd.0 21 tick -7919> 2014-05-19 18:35:56.485652 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7918> 2014-05-19 18:35:56.485655 7fd04f907700 10 osd.0 21 do_waiters -- start -7917> 2014-05-19 18:35:56.485657 7fd04f907700 10 osd.0 21 do_waiters -- finish -7916> 2014-05-19 18:35:56.485689 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7915> 2014-05-19 18:35:56.772846 7fd04e905700 20 KeyValueStore::op_tp worker waiting -7914> 2014-05-19 18:35:56.772850 7fd043ebd700 20 OSD::op_tp worker waiting -7913> 2014-05-19 18:35:56.945054 7fd0396a8700 20 OSD::command_tp worker waiting -7912> 2014-05-19 18:35:56.979579 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:56.979399) v2 ==== 47+0+0 (558342677 0 0) 0x8388a80 con 0x4cb86e0 -7911> 2014-05-19 18:35:56.979631 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:56.979399) v2 -- ?+0 0x798e1c0 con 0x4cb86e0 -7910> 2014-05-19 18:35:56.979663 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7909> 2014-05-19 18:35:56.979698 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:56.979399) v2 ==== 47+0+0 (558342677 0 0) 0x5c05880 con 0x4cb8420 -7908> 2014-05-19 18:35:56.979768 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:56.979399) v2 -- ?+0 0x70cd500 con 0x4cb8420 -7907> 2014-05-19 18:35:56.979798 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7906> 2014-05-19 18:35:57.249612 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7905> 2014-05-19 18:35:57.249631 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7904> 2014-05-19 18:35:57.249638 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x5519340 con 0x3622580 -7903> 2014-05-19 18:35:57.249665 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x55c21c0 con 0x3622840 -7902> 2014-05-19 18:35:57.249702 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x725d880 con 0x3622b00 -7901> 2014-05-19 18:35:57.249717 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x6e2a700 con 0x3622dc0 -7900> 2014-05-19 18:35:57.249731 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x6c4d6c0 con 0x30594a0 -7899> 2014-05-19 18:35:57.249760 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x612c8c0 con 0x3622f20 -7898> 2014-05-19 18:35:57.249775 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x5f32380 con 0x5e7ef20 -7897> 2014-05-19 18:35:57.249798 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x7970540 con 0x5e7f1e0 -7896> 2014-05-19 18:35:57.249830 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x5f7b340 con 0x5e7e2c0 -7895> 2014-05-19 18:35:57.249857 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x77476c0 con 0x5d7bb80 -7894> 2014-05-19 18:35:57.249884 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x5f07340 con 0x5e7ec60 -7893> 2014-05-19 18:35:57.249908 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x59bcfc0 con 0x531b760 -7892> 2014-05-19 18:35:57.249932 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x60f6700 con 0x53bba20 -7891> 2014-05-19 18:35:57.249976 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x72e41c0 con 0x51cac60 -7890> 2014-05-19 18:35:57.250002 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x70a9dc0 con 0x5bd66e0 -7889> 2014-05-19 18:35:57.250027 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:57.249637) v2 -- ?+0 0x74a6700 con 0x5bd6840 -7888> 2014-05-19 18:35:57.250030 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x5f7ac40 con 0x3622b00 -7887> 2014-05-19 18:35:57.250164 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7886> 2014-05-19 18:35:57.250177 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x77468c0 con 0x3622580 -7885> 2014-05-19 18:35:57.250199 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7884> 2014-05-19 18:35:57.250239 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x6b716c0 con 0x3622840 -7883> 2014-05-19 18:35:57.250275 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7882> 2014-05-19 18:35:57.250318 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x6e19c00 con 0x3622dc0 -7881> 2014-05-19 18:35:57.250340 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7880> 2014-05-19 18:35:57.250347 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x3511a40 con 0x5e7ef20 -7879> 2014-05-19 18:35:57.250398 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7878> 2014-05-19 18:35:57.250413 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x77336c0 con 0x5e7f1e0 -7877> 2014-05-19 18:35:57.250442 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7876> 2014-05-19 18:35:57.250453 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x7999880 con 0x5e7e2c0 -7875> 2014-05-19 18:35:57.250474 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7874> 2014-05-19 18:35:57.250483 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x6009500 con 0x5d7bb80 -7873> 2014-05-19 18:35:57.250497 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7872> 2014-05-19 18:35:57.250503 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x6542c40 con 0x5e7ec60 -7871> 2014-05-19 18:35:57.250520 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7870> 2014-05-19 18:35:57.250527 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x692da40 con 0x531b760 -7869> 2014-05-19 18:35:57.250542 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7868> 2014-05-19 18:35:57.250551 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x3438380 con 0x53bba20 -7867> 2014-05-19 18:35:57.250569 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7866> 2014-05-19 18:35:57.250576 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x77e81c0 con 0x51cac60 -7865> 2014-05-19 18:35:57.250591 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7864> 2014-05-19 18:35:57.250599 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x60a16c0 con 0x5bd66e0 -7863> 2014-05-19 18:35:57.250616 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7862> 2014-05-19 18:35:57.250624 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 58 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x71221c0 con 0x5bd6840 -7861> 2014-05-19 18:35:57.250641 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7860> 2014-05-19 18:35:57.250648 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x6c4c540 con 0x30594a0 -7859> 2014-05-19 18:35:57.250665 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7858> 2014-05-19 18:35:57.250672 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:57.249637) v2 ==== 47+0+0 (939910439 0 0) 0x6113880 con 0x3622f20 -7857> 2014-05-19 18:35:57.250690 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7856> 2014-05-19 18:35:57.339333 7fd03c6ae700 20 OSD::op_tp worker waiting -7855> 2014-05-19 18:35:57.339420 7fd041eb9700 20 OSD::op_tp worker waiting -7854> 2014-05-19 18:35:57.339481 7fd03e6b2700 20 OSD::op_tp worker waiting -7853> 2014-05-19 18:35:57.339880 7fd03d6b0700 20 OSD::op_tp worker waiting -7852> 2014-05-19 18:35:57.340062 7fd0436bc700 20 OSD::op_tp worker waiting -7851> 2014-05-19 18:35:57.340159 7fd03bead700 20 OSD::op_tp worker waiting -7850> 2014-05-19 18:35:57.340178 7fd03f6b4700 20 OSD::op_tp worker waiting -7849> 2014-05-19 18:35:57.353889 7fd0446be700 20 OSD::op_tp worker waiting -7848> 2014-05-19 18:35:57.353913 7fd03feb5700 20 OSD::op_tp worker waiting -7847> 2014-05-19 18:35:57.361014 7fd040eb7700 20 OSD::op_tp worker waiting -7846> 2014-05-19 18:35:57.366164 7fd0406b6700 20 OSD::op_tp worker waiting -7845> 2014-05-19 18:35:57.367759 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:02.452587) v2 ==== 47+0+0 (3874309267 0 0) 0x5fc3880 con 0x31e1600 -7844> 2014-05-19 18:35:57.367768 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:02.452587) v2 ==== 47+0+0 (3874309267 0 0) 0x7980380 con 0x61b0160 -7843> 2014-05-19 18:35:57.367806 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:02.452587) v2 -- ?+0 0x8388a80 con 0x31e1600 -7842> 2014-05-19 18:35:57.367885 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7841> 2014-05-19 18:35:57.367929 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:02.452587) v2 -- ?+0 0x5c05880 con 0x61b0160 -7840> 2014-05-19 18:35:57.367976 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7839> 2014-05-19 18:35:57.387820 7fd039ea9700 20 OSD::disk_tp worker waiting -7838> 2014-05-19 18:35:57.391353 7fd03aeab700 20 OSD::op_tp worker waiting -7837> 2014-05-19 18:35:57.391379 7fd03eeb3700 20 OSD::op_tp worker waiting -7836> 2014-05-19 18:35:57.408279 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:02.493123) v2 ==== 47+0+0 (1584364280 0 0) 0x5666a80 con 0x30946e0 -7835> 2014-05-19 18:35:57.408300 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 54 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:02.493123) v2 ==== 47+0+0 (1584364280 0 0) 0x5f0b880 con 0x3094160 -7834> 2014-05-19 18:35:57.408332 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:02.493123) v2 -- ?+0 0x5fc3880 con 0x30946e0 -7833> 2014-05-19 18:35:57.408369 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7832> 2014-05-19 18:35:57.408427 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:02.493123) v2 -- ?+0 0x7980380 con 0x3094160 -7831> 2014-05-19 18:35:57.408465 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7830> 2014-05-19 18:35:57.409252 7fd042ebb700 20 OSD::op_tp worker waiting -7829> 2014-05-19 18:35:57.409274 7fd03deb1700 20 OSD::op_tp worker waiting -7828> 2014-05-19 18:35:57.417816 7fd04e104700 20 KeyValueStore::op_tp worker waiting -7827> 2014-05-19 18:35:57.417817 7fd0426ba700 20 OSD::op_tp worker waiting -7826> 2014-05-19 18:35:57.417853 7fd03b6ac700 20 OSD::op_tp worker waiting -7825> 2014-05-19 18:35:57.417873 7fd03ceaf700 20 OSD::op_tp worker waiting -7824> 2014-05-19 18:35:57.417921 7fd0416b8700 20 OSD::op_tp worker waiting -7823> 2014-05-19 18:35:57.485744 7fd04f907700 5 osd.0 21 tick -7822> 2014-05-19 18:35:57.485761 7fd04f907700 7 osd.0 21 do_mon_report -7821> 2014-05-19 18:35:57.485765 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -7820> 2014-05-19 18:35:57.485769 7fd04f907700 20 osd.0 21 send_pg_stats -7819> 2014-05-19 18:35:57.485777 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7818> 2014-05-19 18:35:57.485781 7fd04f907700 10 osd.0 21 do_waiters -- start -7817> 2014-05-19 18:35:57.485783 7fd04f907700 10 osd.0 21 do_waiters -- finish -7816> 2014-05-19 18:35:57.485821 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7815> 2014-05-19 18:35:58.053914 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:01.110394) v2 ==== 47+0+0 (631673466 0 0) 0x691bdc0 con 0x4cb8dc0 -7814> 2014-05-19 18:35:58.053949 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:01.110394) v2 ==== 47+0+0 (631673466 0 0) 0x73c2000 con 0x4cb9340 -7813> 2014-05-19 18:35:58.053965 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:01.110394) v2 -- ?+0 0x5666a80 con 0x4cb8dc0 -7812> 2014-05-19 18:35:58.053995 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7811> 2014-05-19 18:35:58.054015 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:01.110394) v2 -- ?+0 0x5f0b880 con 0x4cb9340 -7810> 2014-05-19 18:35:58.054046 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7809> 2014-05-19 18:35:58.485865 7fd04f907700 5 osd.0 21 tick -7808> 2014-05-19 18:35:58.485885 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7807> 2014-05-19 18:35:58.485890 7fd04f907700 10 osd.0 21 do_waiters -- start -7806> 2014-05-19 18:35:58.485892 7fd04f907700 10 osd.0 21 do_waiters -- finish -7805> 2014-05-19 18:35:58.485933 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7804> 2014-05-19 18:35:58.772901 7fd04e905700 20 KeyValueStore::op_tp worker waiting -7803> 2014-05-19 18:35:58.772931 7fd043ebd700 20 OSD::op_tp worker waiting -7802> 2014-05-19 18:35:58.945139 7fd0396a8700 20 OSD::command_tp worker waiting -7801> 2014-05-19 18:35:58.950165 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7800> 2014-05-19 18:35:58.950184 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7799> 2014-05-19 18:35:58.950192 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x788a700 con 0x3622580 -7798> 2014-05-19 18:35:58.950217 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x65421c0 con 0x3622840 -7797> 2014-05-19 18:35:58.950257 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x6008e00 con 0x3622b00 -7796> 2014-05-19 18:35:58.950272 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x612dc00 con 0x3622dc0 -7795> 2014-05-19 18:35:58.950294 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x64d2a80 con 0x30594a0 -7794> 2014-05-19 18:35:58.950323 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x5f328c0 con 0x3622f20 -7793> 2014-05-19 18:35:58.950353 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x6c4d500 con 0x5e7ef20 -7792> 2014-05-19 18:35:58.950383 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x7971dc0 con 0x5e7f1e0 -7791> 2014-05-19 18:35:58.950407 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x55c2380 con 0x5e7e2c0 -7790> 2014-05-19 18:35:58.950433 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x7747180 con 0x5d7bb80 -7789> 2014-05-19 18:35:58.950459 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x72e48c0 con 0x5e7ec60 -7788> 2014-05-19 18:35:58.950485 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x68ab500 con 0x531b760 -7787> 2014-05-19 18:35:58.950511 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x788b180 con 0x53bba20 -7786> 2014-05-19 18:35:58.950557 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x6543a40 con 0x51cac60 -7785> 2014-05-19 18:35:58.950584 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x73c3a40 con 0x5bd66e0 -7784> 2014-05-19 18:35:58.950578 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x51a6a80 con 0x3622580 -7783> 2014-05-19 18:35:58.950608 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:35:58.950190) v2 -- ?+0 0x59bcc40 con 0x5bd6840 -7782> 2014-05-19 18:35:58.950654 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7781> 2014-05-19 18:35:58.950676 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x64ae000 con 0x3622dc0 -7780> 2014-05-19 18:35:58.950719 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7779> 2014-05-19 18:35:58.950732 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x6b70380 con 0x3622840 -7778> 2014-05-19 18:35:58.950756 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7777> 2014-05-19 18:35:58.950767 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x76adc00 con 0x3622b00 -7776> 2014-05-19 18:35:58.950785 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7775> 2014-05-19 18:35:58.950803 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x3511500 con 0x5e7ef20 -7774> 2014-05-19 18:35:58.950822 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7773> 2014-05-19 18:35:58.950832 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x7998a80 con 0x5e7e2c0 -7772> 2014-05-19 18:35:58.950851 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7771> 2014-05-19 18:35:58.950859 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x60096c0 con 0x5d7bb80 -7770> 2014-05-19 18:35:58.950874 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7769> 2014-05-19 18:35:58.950915 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x6efefc0 con 0x3622f20 -7768> 2014-05-19 18:35:58.950981 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7767> 2014-05-19 18:35:58.950994 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x5519880 con 0x5e7ec60 -7766> 2014-05-19 18:35:58.951013 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7765> 2014-05-19 18:35:58.951019 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x3438a80 con 0x53bba20 -7764> 2014-05-19 18:35:58.951033 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7763> 2014-05-19 18:35:58.951044 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x73c0000 con 0x51cac60 -7762> 2014-05-19 18:35:58.951061 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7761> 2014-05-19 18:35:58.951069 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x60a0540 con 0x5bd66e0 -7760> 2014-05-19 18:35:58.951083 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7759> 2014-05-19 18:35:58.951093 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 59 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x36f7880 con 0x5bd6840 -7758> 2014-05-19 18:35:58.951111 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7757> 2014-05-19 18:35:58.951119 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x7732540 con 0x5e7f1e0 -7756> 2014-05-19 18:35:58.951134 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7755> 2014-05-19 18:35:58.951139 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x692cc40 con 0x531b760 -7754> 2014-05-19 18:35:58.951155 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7753> 2014-05-19 18:35:58.951162 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:35:58.950190) v2 ==== 47+0+0 (2038631034 0 0) 0x6c4da40 con 0x30594a0 -7752> 2014-05-19 18:35:58.951177 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7751> 2014-05-19 18:35:59.015890 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:02.072370) v2 ==== 47+0+0 (1051271202 0 0) 0x71cb180 con 0x618fe40 -7750> 2014-05-19 18:35:59.015887 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:02.072370) v2 ==== 47+0+0 (1051271202 0 0) 0x5beba40 con 0x61ae420 -7749> 2014-05-19 18:35:59.015938 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:02.072370) v2 -- ?+0 0x73c2000 con 0x618fe40 -7748> 2014-05-19 18:35:59.015978 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7747> 2014-05-19 18:35:59.016040 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:02.072370) v2 -- ?+0 0x691bdc0 con 0x61ae420 -7746> 2014-05-19 18:35:59.016083 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7745> 2014-05-19 18:35:59.165152 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:59.164957) v2 ==== 47+0+0 (4157363546 0 0) 0x5dff880 con 0x4cb8160 -7744> 2014-05-19 18:35:59.165208 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:59.164957) v2 -- ?+0 0x5beba40 con 0x4cb8160 -7743> 2014-05-19 18:35:59.165166 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:35:59.164957) v2 ==== 47+0+0 (4157363546 0 0) 0x669ee00 con 0x4cb82c0 -7742> 2014-05-19 18:35:59.165312 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7741> 2014-05-19 18:35:59.165333 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:35:59.164957) v2 -- ?+0 0x71cb180 con 0x4cb82c0 -7740> 2014-05-19 18:35:59.165366 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7739> 2014-05-19 18:35:59.339411 7fd03c6ae700 20 OSD::op_tp worker waiting -7738> 2014-05-19 18:35:59.339494 7fd041eb9700 20 OSD::op_tp worker waiting -7737> 2014-05-19 18:35:59.339560 7fd03e6b2700 20 OSD::op_tp worker waiting -7736> 2014-05-19 18:35:59.339961 7fd03d6b0700 20 OSD::op_tp worker waiting -7735> 2014-05-19 18:35:59.340140 7fd0436bc700 20 OSD::op_tp worker waiting -7734> 2014-05-19 18:35:59.340242 7fd03bead700 20 OSD::op_tp worker waiting -7733> 2014-05-19 18:35:59.340266 7fd03f6b4700 20 OSD::op_tp worker waiting -7732> 2014-05-19 18:35:59.353986 7fd0446be700 20 OSD::op_tp worker waiting -7731> 2014-05-19 18:35:59.354012 7fd03feb5700 20 OSD::op_tp worker waiting -7730> 2014-05-19 18:35:59.361111 7fd040eb7700 20 OSD::op_tp worker waiting -7729> 2014-05-19 18:35:59.366261 7fd0406b6700 20 OSD::op_tp worker waiting -7728> 2014-05-19 18:35:59.387916 7fd039ea9700 20 OSD::disk_tp worker waiting -7727> 2014-05-19 18:35:59.391451 7fd03aeab700 20 OSD::op_tp worker waiting -7726> 2014-05-19 18:35:59.391476 7fd03eeb3700 20 OSD::op_tp worker waiting -7725> 2014-05-19 18:35:59.409350 7fd042ebb700 20 OSD::op_tp worker waiting -7724> 2014-05-19 18:35:59.409376 7fd03deb1700 20 OSD::op_tp worker waiting -7723> 2014-05-19 18:35:59.417919 7fd04e104700 20 KeyValueStore::op_tp worker waiting -7722> 2014-05-19 18:35:59.417922 7fd0426ba700 20 OSD::op_tp worker waiting -7721> 2014-05-19 18:35:59.417961 7fd03b6ac700 20 OSD::op_tp worker waiting -7720> 2014-05-19 18:35:59.417985 7fd0416b8700 20 OSD::op_tp worker waiting -7719> 2014-05-19 18:35:59.418014 7fd03ceaf700 20 OSD::op_tp worker waiting -7718> 2014-05-19 18:35:59.485980 7fd04f907700 5 osd.0 21 tick -7717> 2014-05-19 18:35:59.486028 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.11 < max 0.5 = yes -7716> 2014-05-19 18:35:59.486042 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -7715> 2014-05-19 18:35:59.486049 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7714> 2014-05-19 18:35:59.486045 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 202.21 < min (86400 seconds) -7713> 2014-05-19 18:35:59.486063 7fd04f907700 20 osd.0 21 sched_scrub done -7712> 2014-05-19 18:35:59.486067 7fd04f907700 10 osd.0 21 do_waiters -- start -7711> 2014-05-19 18:35:59.486068 7fd04f907700 10 osd.0 21 do_waiters -- finish -7710> 2014-05-19 18:35:59.906770 7fd0456c0700 10 monclient: tick -7709> 2014-05-19 18:35:59.906783 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:35:29.906781) -7708> 2014-05-19 18:35:59.906800 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:35:59.906799; renew after: 2014-05-19 18:38:19.906664) -- no -7707> 2014-05-19 18:35:59.983572 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:05.068478) v2 ==== 47+0+0 (2309195190 0 0) 0x78b9a40 con 0x5f5b760 -7706> 2014-05-19 18:35:59.983618 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:05.068478) v2 -- ?+0 0x669ee00 con 0x5f5b760 -7705> 2014-05-19 18:35:59.983653 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7704> 2014-05-19 18:35:59.983679 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:05.068478) v2 ==== 47+0+0 (2309195190 0 0) 0x55c88c0 con 0x5b21600 -7703> 2014-05-19 18:35:59.983714 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:05.068478) v2 -- ?+0 0x5dff880 con 0x5b21600 -7702> 2014-05-19 18:35:59.983743 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7701> 2014-05-19 18:35:59.985613 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:03.042133) v2 ==== 47+0+0 (918869098 0 0) 0x798fc00 con 0x61ae160 -7700> 2014-05-19 18:35:59.985665 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.042133) v2 -- ?+0 0x55c88c0 con 0x61ae160 -7699> 2014-05-19 18:35:59.985700 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7698> 2014-05-19 18:35:59.985673 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:03.042133) v2 ==== 47+0+0 (918869098 0 0) 0x66f9dc0 con 0x61b0420 -7697> 2014-05-19 18:35:59.985720 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.042133) v2 -- ?+0 0x78b9a40 con 0x61b0420 -7696> 2014-05-19 18:35:59.985752 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7695> 2014-05-19 18:36:00.486151 7fd04f907700 5 osd.0 21 tick -7694> 2014-05-19 18:36:00.486174 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7693> 2014-05-19 18:36:00.486181 7fd04f907700 10 osd.0 21 do_waiters -- start -7692> 2014-05-19 18:36:00.486185 7fd04f907700 10 osd.0 21 do_waiters -- finish -7691> 2014-05-19 18:36:00.486212 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7690> 2014-05-19 18:36:00.716326 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:03.772829) v2 ==== 47+0+0 (99101421 0 0) 0x71cb340 con 0x618fe40 -7689> 2014-05-19 18:36:00.716335 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:03.772829) v2 ==== 47+0+0 (99101421 0 0) 0x5bea380 con 0x61ae420 -7688> 2014-05-19 18:36:00.716366 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.772829) v2 -- ?+0 0x66f9dc0 con 0x618fe40 -7687> 2014-05-19 18:36:00.716409 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7686> 2014-05-19 18:36:00.716434 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.772829) v2 -- ?+0 0x798fc00 con 0x61ae420 -7685> 2014-05-19 18:36:00.716495 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7684> 2014-05-19 18:36:00.772977 7fd04e905700 20 KeyValueStore::op_tp worker waiting -7683> 2014-05-19 18:36:00.773009 7fd043ebd700 20 OSD::op_tp worker waiting -7682> 2014-05-19 18:36:00.865673 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:00.865501) v2 ==== 47+0+0 (2473914489 0 0) 0x5dfe8c0 con 0x4cb8160 -7681> 2014-05-19 18:36:00.865721 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:00.865501) v2 -- ?+0 0x5bea380 con 0x4cb8160 -7680> 2014-05-19 18:36:00.865720 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:00.865501) v2 ==== 47+0+0 (2473914489 0 0) 0x669f340 con 0x4cb82c0 -7679> 2014-05-19 18:36:00.865753 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7678> 2014-05-19 18:36:00.865795 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:00.865501) v2 -- ?+0 0x71cb340 con 0x4cb82c0 -7677> 2014-05-19 18:36:00.865883 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7676> 2014-05-19 18:36:00.868072 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:05.953119) v2 ==== 47+0+0 (2275077706 0 0) 0x5fc3180 con 0x31e1600 -7675> 2014-05-19 18:36:00.868127 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:05.953119) v2 -- ?+0 0x5dfe8c0 con 0x31e1600 -7674> 2014-05-19 18:36:00.868164 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7673> 2014-05-19 18:36:00.868171 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:05.953119) v2 ==== 47+0+0 (2275077706 0 0) 0x503f880 con 0x61b0160 -7672> 2014-05-19 18:36:00.868215 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:05.953119) v2 -- ?+0 0x669f340 con 0x61b0160 -7671> 2014-05-19 18:36:00.868254 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7670> 2014-05-19 18:36:00.908553 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:05.993563) v2 ==== 47+0+0 (2075704679 0 0) 0x7746540 con 0x30946e0 -7669> 2014-05-19 18:36:00.908562 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 55 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:05.993563) v2 ==== 47+0+0 (2075704679 0 0) 0x5b281c0 con 0x3094160 -7668> 2014-05-19 18:36:00.908624 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:05.993563) v2 -- ?+0 0x5fc3180 con 0x30946e0 -7667> 2014-05-19 18:36:00.908669 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7666> 2014-05-19 18:36:00.908694 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:05.993563) v2 -- ?+0 0x503f880 con 0x3094160 -7665> 2014-05-19 18:36:00.908732 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7664> 2014-05-19 18:36:00.945235 7fd0396a8700 20 OSD::command_tp worker waiting -7663> 2014-05-19 18:36:01.080116 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:01.079940) v2 ==== 47+0+0 (2961999957 0 0) 0x5deb500 con 0x4cb86e0 -7662> 2014-05-19 18:36:01.080182 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:01.079940) v2 ==== 47+0+0 (2961999957 0 0) 0x5c05500 con 0x4cb8420 -7661> 2014-05-19 18:36:01.080188 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:01.079940) v2 -- ?+0 0x7746540 con 0x4cb86e0 -7660> 2014-05-19 18:36:01.080311 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7659> 2014-05-19 18:36:01.080374 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:01.079940) v2 -- ?+0 0x5b281c0 con 0x4cb8420 -7658> 2014-05-19 18:36:01.080428 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7657> 2014-05-19 18:36:01.339504 7fd03c6ae700 20 OSD::op_tp worker waiting -7656> 2014-05-19 18:36:01.339579 7fd041eb9700 20 OSD::op_tp worker waiting -7655> 2014-05-19 18:36:01.339638 7fd03e6b2700 20 OSD::op_tp worker waiting -7654> 2014-05-19 18:36:01.340041 7fd03d6b0700 20 OSD::op_tp worker waiting -7653> 2014-05-19 18:36:01.340216 7fd0436bc700 20 OSD::op_tp worker waiting -7652> 2014-05-19 18:36:01.340321 7fd03bead700 20 OSD::op_tp worker waiting -7651> 2014-05-19 18:36:01.340342 7fd03f6b4700 20 OSD::op_tp worker waiting -7650> 2014-05-19 18:36:01.354084 7fd0446be700 20 OSD::op_tp worker waiting -7649> 2014-05-19 18:36:01.354108 7fd03feb5700 20 OSD::op_tp worker waiting -7648> 2014-05-19 18:36:01.361208 7fd040eb7700 20 OSD::op_tp worker waiting -7647> 2014-05-19 18:36:01.366358 7fd0406b6700 20 OSD::op_tp worker waiting -7646> 2014-05-19 18:36:01.388013 7fd039ea9700 20 OSD::disk_tp worker waiting -7645> 2014-05-19 18:36:01.391550 7fd03aeab700 20 OSD::op_tp worker waiting -7644> 2014-05-19 18:36:01.391574 7fd03eeb3700 20 OSD::op_tp worker waiting -7643> 2014-05-19 18:36:01.409450 7fd042ebb700 20 OSD::op_tp worker waiting -7642> 2014-05-19 18:36:01.409474 7fd03deb1700 20 OSD::op_tp worker waiting -7641> 2014-05-19 18:36:01.418024 7fd04e104700 20 KeyValueStore::op_tp worker waiting -7640> 2014-05-19 18:36:01.418030 7fd0426ba700 20 OSD::op_tp worker waiting -7639> 2014-05-19 18:36:01.418053 7fd03b6ac700 20 OSD::op_tp worker waiting -7638> 2014-05-19 18:36:01.418067 7fd0416b8700 20 OSD::op_tp worker waiting -7637> 2014-05-19 18:36:01.418092 7fd03ceaf700 20 OSD::op_tp worker waiting -7636> 2014-05-19 18:36:01.486273 7fd04f907700 5 osd.0 21 tick -7635> 2014-05-19 18:36:01.486325 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7634> 2014-05-19 18:36:01.486321 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.11 < max 0.5 = yes -7633> 2014-05-19 18:36:01.486335 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -7632> 2014-05-19 18:36:01.486338 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 204.21 < min (86400 seconds) -7631> 2014-05-19 18:36:01.486356 7fd04f907700 20 osd.0 21 sched_scrub done -7630> 2014-05-19 18:36:01.486360 7fd04f907700 10 osd.0 21 do_waiters -- start -7629> 2014-05-19 18:36:01.486361 7fd04f907700 10 osd.0 21 do_waiters -- finish -7628> 2014-05-19 18:36:02.154464 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:05.210942) v2 ==== 47+0+0 (2151551294 0 0) 0x691a700 con 0x4cb8dc0 -7627> 2014-05-19 18:36:02.154535 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:05.210942) v2 -- ?+0 0x5deb500 con 0x4cb8dc0 -7626> 2014-05-19 18:36:02.154532 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:05.210942) v2 ==== 47+0+0 (2151551294 0 0) 0x68aa1c0 con 0x4cb9340 -7625> 2014-05-19 18:36:02.154578 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7624> 2014-05-19 18:36:02.154595 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:05.210942) v2 -- ?+0 0x5c05500 con 0x4cb9340 -7623> 2014-05-19 18:36:02.154623 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7622> 2014-05-19 18:36:02.486443 7fd04f907700 5 osd.0 21 tick -7621> 2014-05-19 18:36:02.486461 7fd04f907700 7 osd.0 21 do_mon_report -7620> 2014-05-19 18:36:02.486467 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -7619> 2014-05-19 18:36:02.486473 7fd04f907700 20 osd.0 21 send_pg_stats -7618> 2014-05-19 18:36:02.486522 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7617> 2014-05-19 18:36:02.486515 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.11 < max 0.5 = yes -7616> 2014-05-19 18:36:02.486533 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -7615> 2014-05-19 18:36:02.486538 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 205.211 < min (86400 seconds) -7614> 2014-05-19 18:36:02.486575 7fd04f907700 20 osd.0 21 sched_scrub done -7613> 2014-05-19 18:36:02.486581 7fd04f907700 10 osd.0 21 do_waiters -- start -7612> 2014-05-19 18:36:02.486584 7fd04f907700 10 osd.0 21 do_waiters -- finish -7611> 2014-05-19 18:36:02.773043 7fd04e905700 20 KeyValueStore::op_tp worker waiting -7610> 2014-05-19 18:36:02.773122 7fd043ebd700 20 OSD::op_tp worker waiting -7609> 2014-05-19 18:36:02.780649 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:02.780448) v2 ==== 47+0+0 (1320829869 0 0) 0x5dea8c0 con 0x4cb86e0 -7608> 2014-05-19 18:36:02.780650 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:02.780448) v2 ==== 47+0+0 (1320829869 0 0) 0x5c05340 con 0x4cb8420 -7607> 2014-05-19 18:36:02.780705 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:02.780448) v2 -- ?+0 0x691a700 con 0x4cb86e0 -7606> 2014-05-19 18:36:02.780760 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7605> 2014-05-19 18:36:02.780807 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:02.780448) v2 -- ?+0 0x68aa1c0 con 0x4cb8420 -7604> 2014-05-19 18:36:02.780852 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7603> 2014-05-19 18:36:02.945321 7fd0396a8700 20 OSD::command_tp worker waiting -7602> 2014-05-19 18:36:03.339589 7fd03c6ae700 20 OSD::op_tp worker waiting -7601> 2014-05-19 18:36:03.339656 7fd041eb9700 20 OSD::op_tp worker waiting -7600> 2014-05-19 18:36:03.339717 7fd03e6b2700 20 OSD::op_tp worker waiting -7599> 2014-05-19 18:36:03.340122 7fd03d6b0700 20 OSD::op_tp worker waiting -7598> 2014-05-19 18:36:03.340302 7fd0436bc700 20 OSD::op_tp worker waiting -7597> 2014-05-19 18:36:03.340404 7fd03bead700 20 OSD::op_tp worker waiting -7596> 2014-05-19 18:36:03.340432 7fd03f6b4700 20 OSD::op_tp worker waiting -7595> 2014-05-19 18:36:03.354183 7fd0446be700 20 OSD::op_tp worker waiting -7594> 2014-05-19 18:36:03.354206 7fd03feb5700 20 OSD::op_tp worker waiting -7593> 2014-05-19 18:36:03.361300 7fd040eb7700 20 OSD::op_tp worker waiting -7592> 2014-05-19 18:36:03.366456 7fd0406b6700 20 OSD::op_tp worker waiting -7591> 2014-05-19 18:36:03.388107 7fd039ea9700 20 OSD::disk_tp worker waiting -7590> 2014-05-19 18:36:03.391650 7fd03aeab700 20 OSD::op_tp worker waiting -7589> 2014-05-19 18:36:03.391677 7fd03eeb3700 20 OSD::op_tp worker waiting -7588> 2014-05-19 18:36:03.409550 7fd042ebb700 20 OSD::op_tp worker waiting -7587> 2014-05-19 18:36:03.409573 7fd03deb1700 20 OSD::op_tp worker waiting -7586> 2014-05-19 18:36:03.418109 7fd04e104700 20 KeyValueStore::op_tp worker waiting -7585> 2014-05-19 18:36:03.418113 7fd0426ba700 20 OSD::op_tp worker waiting -7584> 2014-05-19 18:36:03.418141 7fd03b6ac700 20 OSD::op_tp worker waiting -7583> 2014-05-19 18:36:03.418164 7fd0416b8700 20 OSD::op_tp worker waiting -7582> 2014-05-19 18:36:03.418184 7fd03ceaf700 20 OSD::op_tp worker waiting -7581> 2014-05-19 18:36:03.486672 7fd04f907700 5 osd.0 21 tick -7580> 2014-05-19 18:36:03.486692 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7579> 2014-05-19 18:36:03.486697 7fd04f907700 10 osd.0 21 do_waiters -- start -7578> 2014-05-19 18:36:03.486699 7fd04f907700 10 osd.0 21 do_waiters -- finish -7577> 2014-05-19 18:36:03.486722 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7576> 2014-05-19 18:36:03.650744 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7575> 2014-05-19 18:36:03.650763 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7574> 2014-05-19 18:36:03.650775 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x5f06540 con 0x3622580 -7573> 2014-05-19 18:36:03.650807 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x8388700 con 0x3622840 -7572> 2014-05-19 18:36:03.650834 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x6c4ca80 con 0x3622b00 -7571> 2014-05-19 18:36:03.650863 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x60081c0 con 0x3622dc0 -7570> 2014-05-19 18:36:03.650890 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x77336c0 con 0x30594a0 -7569> 2014-05-19 18:36:03.650917 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x3511a40 con 0x3622f20 -7568> 2014-05-19 18:36:03.650943 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x6e19c00 con 0x5e7ef20 -7567> 2014-05-19 18:36:03.650969 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x6b716c0 con 0x5e7f1e0 -7566> 2014-05-19 18:36:03.650994 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x77468c0 con 0x5e7e2c0 -7565> 2014-05-19 18:36:03.651019 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x5f7ac40 con 0x5d7bb80 -7564> 2014-05-19 18:36:03.651041 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x788a540 con 0x5e7ec60 -7563> 2014-05-19 18:36:03.651065 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x6543180 con 0x531b760 -7562> 2014-05-19 18:36:03.651089 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x7931a40 con 0x53bba20 -7561> 2014-05-19 18:36:03.651115 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x59bddc0 con 0x51cac60 -7560> 2014-05-19 18:36:03.651140 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x7122a80 con 0x5bd66e0 -7559> 2014-05-19 18:36:03.651167 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:03.650773) v2 -- ?+0 0x60a0700 con 0x5bd6840 -7558> 2014-05-19 18:36:03.651171 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x6b71dc0 con 0x3622840 -7557> 2014-05-19 18:36:03.651218 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7556> 2014-05-19 18:36:03.651236 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x64afc00 con 0x3622dc0 -7555> 2014-05-19 18:36:03.651264 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7554> 2014-05-19 18:36:03.651280 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x76acfc0 con 0x3622b00 -7553> 2014-05-19 18:36:03.651313 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7552> 2014-05-19 18:36:03.651328 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x51a6540 con 0x3622580 -7551> 2014-05-19 18:36:03.651378 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7550> 2014-05-19 18:36:03.651388 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x3511180 con 0x5e7ef20 -7549> 2014-05-19 18:36:03.651407 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7548> 2014-05-19 18:36:03.651413 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x682f340 con 0x30594a0 -7547> 2014-05-19 18:36:03.651427 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7546> 2014-05-19 18:36:03.651451 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x6009340 con 0x5d7bb80 -7545> 2014-05-19 18:36:03.651483 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7544> 2014-05-19 18:36:03.651564 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x60f61c0 con 0x3622f20 -7543> 2014-05-19 18:36:03.651599 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7542> 2014-05-19 18:36:03.651626 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x7733dc0 con 0x5e7f1e0 -7541> 2014-05-19 18:36:03.651651 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7540> 2014-05-19 18:36:03.651659 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x7999dc0 con 0x5e7e2c0 -7539> 2014-05-19 18:36:03.651677 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7538> 2014-05-19 18:36:03.651688 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x5519dc0 con 0x5e7ec60 -7537> 2014-05-19 18:36:03.651705 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7536> 2014-05-19 18:36:03.651713 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x692d500 con 0x531b760 -7535> 2014-05-19 18:36:03.651728 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7534> 2014-05-19 18:36:03.651737 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x6c4b180 con 0x53bba20 -7533> 2014-05-19 18:36:03.651754 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7532> 2014-05-19 18:36:03.651761 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x73c0540 con 0x51cac60 -7531> 2014-05-19 18:36:03.651776 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7530> 2014-05-19 18:36:03.651783 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x60a0e00 con 0x5bd66e0 -7529> 2014-05-19 18:36:03.651797 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7528> 2014-05-19 18:36:03.651805 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 60 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:03.650773) v2 ==== 47+0+0 (4125855931 0 0) 0x36f6000 con 0x5bd6840 -7527> 2014-05-19 18:36:03.651823 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7526> 2014-05-19 18:36:03.854875 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:06.911477) v2 ==== 47+0+0 (451231714 0 0) 0x5b28540 con 0x4cb9340 -7525> 2014-05-19 18:36:03.854893 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:06.911477) v2 ==== 47+0+0 (451231714 0 0) 0x691b340 con 0x4cb8dc0 -7524> 2014-05-19 18:36:03.854928 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:06.911477) v2 -- ?+0 0x5c05340 con 0x4cb9340 -7523> 2014-05-19 18:36:03.855006 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7522> 2014-05-19 18:36:03.855023 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:06.911477) v2 -- ?+0 0x5dea8c0 con 0x4cb8dc0 -7521> 2014-05-19 18:36:03.855057 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7520> 2014-05-19 18:36:04.086378 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:07.142944) v2 ==== 47+0+0 (420589944 0 0) 0x798e700 con 0x61ae160 -7519> 2014-05-19 18:36:04.086422 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:07.142944) v2 -- ?+0 0x691b340 con 0x61ae160 -7518> 2014-05-19 18:36:04.086456 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7517> 2014-05-19 18:36:04.086445 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:07.142944) v2 ==== 47+0+0 (420589944 0 0) 0x66f9500 con 0x61b0420 -7516> 2014-05-19 18:36:04.086496 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:07.142944) v2 -- ?+0 0x5b28540 con 0x61b0420 -7515> 2014-05-19 18:36:04.086529 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7514> 2014-05-19 18:36:04.486785 7fd04f907700 5 osd.0 21 tick -7513> 2014-05-19 18:36:04.486834 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.1 < max 0.5 = yes -7512> 2014-05-19 18:36:04.486847 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -7511> 2014-05-19 18:36:04.486854 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7510> 2014-05-19 18:36:04.486850 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 207.211 < min (86400 seconds) -7509> 2014-05-19 18:36:04.486870 7fd04f907700 20 osd.0 21 sched_scrub done -7508> 2014-05-19 18:36:04.486873 7fd04f907700 10 osd.0 21 do_waiters -- start -7507> 2014-05-19 18:36:04.486875 7fd04f907700 10 osd.0 21 do_waiters -- finish -7506> 2014-05-19 18:36:04.773100 7fd04e905700 20 KeyValueStore::op_tp worker waiting -7505> 2014-05-19 18:36:04.773206 7fd043ebd700 20 OSD::op_tp worker waiting -7504> 2014-05-19 18:36:04.945405 7fd0396a8700 20 OSD::command_tp worker waiting -7503> 2014-05-19 18:36:04.968618 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:10.053707) v2 ==== 47+0+0 (1464918061 0 0) 0x503e000 con 0x61b0160 -7502> 2014-05-19 18:36:04.968644 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:10.053707) v2 ==== 47+0+0 (1464918061 0 0) 0x772cc40 con 0x31e1600 -7501> 2014-05-19 18:36:04.968668 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.053707) v2 -- ?+0 0x66f9500 con 0x61b0160 -7500> 2014-05-19 18:36:04.968702 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7499> 2014-05-19 18:36:04.968740 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.053707) v2 -- ?+0 0x798e700 con 0x31e1600 -7498> 2014-05-19 18:36:04.968795 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7497> 2014-05-19 18:36:05.283850 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:10.369012) v2 ==== 47+0+0 (2716561906 0 0) 0x55c9a40 con 0x5b21600 -7496> 2014-05-19 18:36:05.283886 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:10.369012) v2 ==== 47+0+0 (2716561906 0 0) 0x73c16c0 con 0x5f5b760 -7495> 2014-05-19 18:36:05.283898 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.369012) v2 -- ?+0 0x772cc40 con 0x5b21600 -7494> 2014-05-19 18:36:05.283930 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7493> 2014-05-19 18:36:05.283985 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.369012) v2 -- ?+0 0x503e000 con 0x5f5b760 -7492> 2014-05-19 18:36:05.284029 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7491> 2014-05-19 18:36:05.339683 7fd03c6ae700 20 OSD::op_tp worker waiting -7490> 2014-05-19 18:36:05.339730 7fd041eb9700 20 OSD::op_tp worker waiting -7489> 2014-05-19 18:36:05.339794 7fd03e6b2700 20 OSD::op_tp worker waiting -7488> 2014-05-19 18:36:05.340203 7fd03d6b0700 20 OSD::op_tp worker waiting -7487> 2014-05-19 18:36:05.340384 7fd0436bc700 20 OSD::op_tp worker waiting -7486> 2014-05-19 18:36:05.340488 7fd03bead700 20 OSD::op_tp worker waiting -7485> 2014-05-19 18:36:05.340509 7fd03f6b4700 20 OSD::op_tp worker waiting -7484> 2014-05-19 18:36:05.354282 7fd0446be700 20 OSD::op_tp worker waiting -7483> 2014-05-19 18:36:05.354307 7fd03feb5700 20 OSD::op_tp worker waiting -7482> 2014-05-19 18:36:05.361381 7fd040eb7700 20 OSD::op_tp worker waiting -7481> 2014-05-19 18:36:05.366553 7fd0406b6700 20 OSD::op_tp worker waiting -7480> 2014-05-19 18:36:05.388209 7fd039ea9700 20 OSD::disk_tp worker waiting -7479> 2014-05-19 18:36:05.391749 7fd03aeab700 20 OSD::op_tp worker waiting -7478> 2014-05-19 18:36:05.391775 7fd03eeb3700 20 OSD::op_tp worker waiting -7477> 2014-05-19 18:36:05.409649 7fd042ebb700 20 OSD::op_tp worker waiting -7476> 2014-05-19 18:36:05.409676 7fd03deb1700 20 OSD::op_tp worker waiting -7475> 2014-05-19 18:36:05.416909 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:08.473466) v2 ==== 47+0+0 (915343001 0 0) 0x5bea700 con 0x61ae420 -7474> 2014-05-19 18:36:05.416937 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:08.473466) v2 ==== 47+0+0 (915343001 0 0) 0x71cafc0 con 0x618fe40 -7473> 2014-05-19 18:36:05.416956 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:08.473466) v2 -- ?+0 0x55c9a40 con 0x61ae420 -7472> 2014-05-19 18:36:05.417035 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7471> 2014-05-19 18:36:05.417068 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:08.473466) v2 -- ?+0 0x73c16c0 con 0x618fe40 -7470> 2014-05-19 18:36:05.417104 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7469> 2014-05-19 18:36:05.418197 7fd04e104700 20 KeyValueStore::op_tp worker waiting -7468> 2014-05-19 18:36:05.418198 7fd0426ba700 20 OSD::op_tp worker waiting -7467> 2014-05-19 18:36:05.418237 7fd03b6ac700 20 OSD::op_tp worker waiting -7466> 2014-05-19 18:36:05.418257 7fd0416b8700 20 OSD::op_tp worker waiting -7465> 2014-05-19 18:36:05.418286 7fd03ceaf700 20 OSD::op_tp worker waiting -7464> 2014-05-19 18:36:05.486962 7fd04f907700 5 osd.0 21 tick -7463> 2014-05-19 18:36:05.486983 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7462> 2014-05-19 18:36:05.486988 7fd04f907700 10 osd.0 21 do_waiters -- start -7461> 2014-05-19 18:36:05.486990 7fd04f907700 10 osd.0 21 do_waiters -- finish -7460> 2014-05-19 18:36:05.487019 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7459> 2014-05-19 18:36:05.566278 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:05.566077) v2 ==== 47+0+0 (2109216291 0 0) 0x70a9340 con 0x4cb8160 -7458> 2014-05-19 18:36:05.566293 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:05.566077) v2 ==== 47+0+0 (2109216291 0 0) 0x5deaa80 con 0x4cb82c0 -7457> 2014-05-19 18:36:05.566315 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:05.566077) v2 -- ?+0 0x5bea700 con 0x4cb8160 -7456> 2014-05-19 18:36:05.566346 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7455> 2014-05-19 18:36:05.566374 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:05.566077) v2 -- ?+0 0x71cafc0 con 0x4cb82c0 -7454> 2014-05-19 18:36:05.566472 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7453> 2014-05-19 18:36:05.786935 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:08.843488) v2 ==== 47+0+0 (1980706377 0 0) 0x798fdc0 con 0x61ae160 -7452> 2014-05-19 18:36:05.786984 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:08.843488) v2 -- ?+0 0x70a9340 con 0x61ae160 -7451> 2014-05-19 18:36:05.786991 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:08.843488) v2 ==== 47+0+0 (1980706377 0 0) 0x66f9c00 con 0x61b0420 -7450> 2014-05-19 18:36:05.787066 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7449> 2014-05-19 18:36:05.787093 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:08.843488) v2 -- ?+0 0x5deaa80 con 0x61b0420 -7448> 2014-05-19 18:36:05.787147 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7447> 2014-05-19 18:36:06.208842 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:11.294002) v2 ==== 47+0+0 (2183563129 0 0) 0x77e8e00 con 0x3094160 -7446> 2014-05-19 18:36:06.208844 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 56 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:11.294002) v2 ==== 47+0+0 (2183563129 0 0) 0x788b500 con 0x30946e0 -7445> 2014-05-19 18:36:06.208892 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:11.294002) v2 -- ?+0 0x66f9c00 con 0x3094160 -7444> 2014-05-19 18:36:06.208926 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7443> 2014-05-19 18:36:06.208965 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:11.294002) v2 -- ?+0 0x798fdc0 con 0x30946e0 -7442> 2014-05-19 18:36:06.209001 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7441> 2014-05-19 18:36:06.487077 7fd04f907700 5 osd.0 21 tick -7440> 2014-05-19 18:36:06.487097 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7439> 2014-05-19 18:36:06.487102 7fd04f907700 10 osd.0 21 do_waiters -- start -7438> 2014-05-19 18:36:06.487104 7fd04f907700 10 osd.0 21 do_waiters -- finish -7437> 2014-05-19 18:36:06.487145 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7436> 2014-05-19 18:36:06.669062 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:11.754243) v2 ==== 47+0+0 (965037720 0 0) 0x691b6c0 con 0x31e1600 -7435> 2014-05-19 18:36:06.669115 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:11.754243) v2 ==== 47+0+0 (965037720 0 0) 0x75b1c00 con 0x61b0160 -7434> 2014-05-19 18:36:06.669132 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:11.754243) v2 -- ?+0 0x788b500 con 0x31e1600 -7433> 2014-05-19 18:36:06.669216 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7432> 2014-05-19 18:36:06.669255 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:11.754243) v2 -- ?+0 0x77e8e00 con 0x61b0160 -7431> 2014-05-19 18:36:06.669295 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7430> 2014-05-19 18:36:06.773176 7fd04e905700 20 KeyValueStore::op_tp worker waiting -7429> 2014-05-19 18:36:06.773287 7fd043ebd700 20 OSD::op_tp worker waiting -7428> 2014-05-19 18:36:06.945502 7fd0396a8700 20 OSD::command_tp worker waiting -7427> 2014-05-19 18:36:06.984318 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:12.069501) v2 ==== 47+0+0 (431401128 0 0) 0x511c380 con 0x5b21600 -7426> 2014-05-19 18:36:06.984350 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:12.069501) v2 ==== 47+0+0 (431401128 0 0) 0x79e76c0 con 0x5f5b760 -7425> 2014-05-19 18:36:06.984370 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.069501) v2 -- ?+0 0x691b6c0 con 0x5b21600 -7424> 2014-05-19 18:36:06.984402 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7423> 2014-05-19 18:36:06.984459 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.069501) v2 -- ?+0 0x75b1c00 con 0x5f5b760 -7422> 2014-05-19 18:36:06.984507 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7421> 2014-05-19 18:36:07.339778 7fd03c6ae700 20 OSD::op_tp worker waiting -7420> 2014-05-19 18:36:07.339802 7fd041eb9700 20 OSD::op_tp worker waiting -7419> 2014-05-19 18:36:07.339880 7fd03e6b2700 20 OSD::op_tp worker waiting -7418> 2014-05-19 18:36:07.340288 7fd03d6b0700 20 OSD::op_tp worker waiting -7417> 2014-05-19 18:36:07.340466 7fd0436bc700 20 OSD::op_tp worker waiting -7416> 2014-05-19 18:36:07.340570 7fd03bead700 20 OSD::op_tp worker waiting -7415> 2014-05-19 18:36:07.340620 7fd03f6b4700 20 OSD::op_tp worker waiting -7414> 2014-05-19 18:36:07.354380 7fd0446be700 20 OSD::op_tp worker waiting -7413> 2014-05-19 18:36:07.354406 7fd03feb5700 20 OSD::op_tp worker waiting -7412> 2014-05-19 18:36:07.361458 7fd040eb7700 20 OSD::op_tp worker waiting -7411> 2014-05-19 18:36:07.366650 7fd0406b6700 20 OSD::op_tp worker waiting -7410> 2014-05-19 18:36:07.388290 7fd039ea9700 20 OSD::disk_tp worker waiting -7409> 2014-05-19 18:36:07.391850 7fd03aeab700 20 OSD::op_tp worker waiting -7408> 2014-05-19 18:36:07.391878 7fd03eeb3700 20 OSD::op_tp worker waiting -7407> 2014-05-19 18:36:07.409750 7fd042ebb700 20 OSD::op_tp worker waiting -7406> 2014-05-19 18:36:07.409774 7fd03deb1700 20 OSD::op_tp worker waiting -7405> 2014-05-19 18:36:07.418286 7fd04e104700 20 KeyValueStore::op_tp worker waiting -7404> 2014-05-19 18:36:07.418288 7fd0426ba700 20 OSD::op_tp worker waiting -7403> 2014-05-19 18:36:07.418326 7fd03b6ac700 20 OSD::op_tp worker waiting -7402> 2014-05-19 18:36:07.418349 7fd0416b8700 20 OSD::op_tp worker waiting -7401> 2014-05-19 18:36:07.418370 7fd03ceaf700 20 OSD::op_tp worker waiting -7400> 2014-05-19 18:36:07.481184 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:07.481012) v2 ==== 47+0+0 (4033297306 0 0) 0x64ae700 con 0x4cb86e0 -7399> 2014-05-19 18:36:07.481234 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:07.481012) v2 -- ?+0 0x511c380 con 0x4cb86e0 -7398> 2014-05-19 18:36:07.481277 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7397> 2014-05-19 18:36:07.481233 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:07.481012) v2 ==== 47+0+0 (4033297306 0 0) 0x5c47880 con 0x4cb8420 -7396> 2014-05-19 18:36:07.481357 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:07.481012) v2 -- ?+0 0x79e76c0 con 0x4cb8420 -7395> 2014-05-19 18:36:07.481396 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7394> 2014-05-19 18:36:07.487171 7fd04f907700 5 osd.0 21 tick -7393> 2014-05-19 18:36:07.487188 7fd04f907700 7 osd.0 21 do_mon_report -7392> 2014-05-19 18:36:07.487191 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -7391> 2014-05-19 18:36:07.487194 7fd04f907700 20 osd.0 21 send_pg_stats -7390> 2014-05-19 18:36:07.487202 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7389> 2014-05-19 18:36:07.487205 7fd04f907700 10 osd.0 21 do_waiters -- start -7388> 2014-05-19 18:36:07.487207 7fd04f907700 10 osd.0 21 do_waiters -- finish -7387> 2014-05-19 18:36:07.487208 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7386> 2014-05-19 18:36:07.909174 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:12.994457) v2 ==== 47+0+0 (3756166463 0 0) 0x78b8c40 con 0x30946e0 -7385> 2014-05-19 18:36:07.909214 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.994457) v2 -- ?+0 0x64ae700 con 0x30946e0 -7384> 2014-05-19 18:36:07.909206 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 57 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:12.994457) v2 ==== 47+0+0 (3756166463 0 0) 0x669e540 con 0x3094160 -7383> 2014-05-19 18:36:07.909289 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7382> 2014-05-19 18:36:07.909310 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.994457) v2 -- ?+0 0x5c47880 con 0x3094160 -7381> 2014-05-19 18:36:07.909342 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7380> 2014-05-19 18:36:08.487294 7fd04f907700 5 osd.0 21 tick -7379> 2014-05-19 18:36:08.487342 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.09 < max 0.5 = yes -7378> 2014-05-19 18:36:08.487356 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -7377> 2014-05-19 18:36:08.487352 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7376> 2014-05-19 18:36:08.487359 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 211.211 < min (86400 seconds) -7375> 2014-05-19 18:36:08.487378 7fd04f907700 20 osd.0 21 sched_scrub done -7374> 2014-05-19 18:36:08.487381 7fd04f907700 10 osd.0 21 do_waiters -- start -7373> 2014-05-19 18:36:08.487382 7fd04f907700 10 osd.0 21 do_waiters -- finish -7372> 2014-05-19 18:36:08.555644 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:11.612190) v2 ==== 47+0+0 (3592897916 0 0) 0x78b9500 con 0x4cb8dc0 -7371> 2014-05-19 18:36:08.555694 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:11.612190) v2 -- ?+0 0x78b8c40 con 0x4cb8dc0 -7370> 2014-05-19 18:36:08.555728 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7369> 2014-05-19 18:36:08.555703 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:11.612190) v2 ==== 47+0+0 (3592897916 0 0) 0x5e28e00 con 0x4cb9340 -7368> 2014-05-19 18:36:08.555741 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:11.612190) v2 -- ?+0 0x669e540 con 0x4cb9340 -7367> 2014-05-19 18:36:08.555774 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7366> 2014-05-19 18:36:08.773257 7fd04e905700 20 KeyValueStore::op_tp worker waiting -7365> 2014-05-19 18:36:08.773368 7fd043ebd700 20 OSD::op_tp worker waiting -7364> 2014-05-19 18:36:08.945588 7fd0396a8700 20 OSD::command_tp worker waiting -7363> 2014-05-19 18:36:09.339833 7fd03c6ae700 20 OSD::op_tp worker waiting -7362> 2014-05-19 18:36:09.339878 7fd041eb9700 20 OSD::op_tp worker waiting -7361> 2014-05-19 18:36:09.339951 7fd03e6b2700 20 OSD::op_tp worker waiting -7360> 2014-05-19 18:36:09.340368 7fd03d6b0700 20 OSD::op_tp worker waiting -7359> 2014-05-19 18:36:09.340504 7fd0436bc700 20 OSD::op_tp worker waiting -7358> 2014-05-19 18:36:09.340670 7fd03bead700 20 OSD::op_tp worker waiting -7357> 2014-05-19 18:36:09.340692 7fd03f6b4700 20 OSD::op_tp worker waiting -7356> 2014-05-19 18:36:09.354479 7fd0446be700 20 OSD::op_tp worker waiting -7355> 2014-05-19 18:36:09.354502 7fd03feb5700 20 OSD::op_tp worker waiting -7354> 2014-05-19 18:36:09.361499 7fd040eb7700 20 OSD::op_tp worker waiting -7353> 2014-05-19 18:36:09.366747 7fd0406b6700 20 OSD::op_tp worker waiting -7352> 2014-05-19 18:36:09.388373 7fd039ea9700 20 OSD::disk_tp worker waiting -7351> 2014-05-19 18:36:09.391950 7fd03aeab700 20 OSD::op_tp worker waiting -7350> 2014-05-19 18:36:09.391976 7fd03eeb3700 20 OSD::op_tp worker waiting -7349> 2014-05-19 18:36:09.409849 7fd042ebb700 20 OSD::op_tp worker waiting -7348> 2014-05-19 18:36:09.409872 7fd03deb1700 20 OSD::op_tp worker waiting -7347> 2014-05-19 18:36:09.418394 7fd04e104700 20 KeyValueStore::op_tp worker waiting -7346> 2014-05-19 18:36:09.418395 7fd0426ba700 20 OSD::op_tp worker waiting -7345> 2014-05-19 18:36:09.418429 7fd03b6ac700 20 OSD::op_tp worker waiting -7344> 2014-05-19 18:36:09.418450 7fd0416b8700 20 OSD::op_tp worker waiting -7343> 2014-05-19 18:36:09.418508 7fd03ceaf700 20 OSD::op_tp worker waiting -7342> 2014-05-19 18:36:09.487470 7fd04f907700 5 osd.0 21 tick -7341> 2014-05-19 18:36:09.487491 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7340> 2014-05-19 18:36:09.487495 7fd04f907700 10 osd.0 21 do_waiters -- start -7339> 2014-05-19 18:36:09.487498 7fd04f907700 10 osd.0 21 do_waiters -- finish -7338> 2014-05-19 18:36:09.487539 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7337> 2014-05-19 18:36:09.551314 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7336> 2014-05-19 18:36:09.551332 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7335> 2014-05-19 18:36:09.551340 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x77e8000 con 0x3622580 -7334> 2014-05-19 18:36:09.551367 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x3438700 con 0x3622840 -7333> 2014-05-19 18:36:09.551404 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x6c4d180 con 0x3622b00 -7332> 2014-05-19 18:36:09.551419 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x6009180 con 0x3622dc0 -7331> 2014-05-19 18:36:09.551432 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x612ce00 con 0x30594a0 -7330> 2014-05-19 18:36:09.551456 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x6504700 con 0x3622f20 -7329> 2014-05-19 18:36:09.551470 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x5f7a8c0 con 0x5e7ef20 -7328> 2014-05-19 18:36:09.551491 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x6b70540 con 0x5e7f1e0 -7327> 2014-05-19 18:36:09.551515 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x7746c40 con 0x5e7e2c0 -7326> 2014-05-19 18:36:09.551545 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x7971500 con 0x5d7bb80 -7325> 2014-05-19 18:36:09.551571 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x7930540 con 0x5e7ec60 -7324> 2014-05-19 18:36:09.551594 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x6504e00 con 0x531b760 -7323> 2014-05-19 18:36:09.551620 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x59bd500 con 0x53bba20 -7322> 2014-05-19 18:36:09.551666 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x71236c0 con 0x51cac60 -7321> 2014-05-19 18:36:09.551699 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x60a1a40 con 0x5bd66e0 -7320> 2014-05-19 18:36:09.551727 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:09.551338) v2 -- ?+0 0x70a81c0 con 0x5bd6840 -7319> 2014-05-19 18:36:09.551736 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x51a7dc0 con 0x3622580 -7318> 2014-05-19 18:36:09.551781 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7317> 2014-05-19 18:36:09.551797 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x64ae8c0 con 0x3622dc0 -7316> 2014-05-19 18:36:09.551826 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7315> 2014-05-19 18:36:09.551845 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x76ac000 con 0x3622b00 -7314> 2014-05-19 18:36:09.551878 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7313> 2014-05-19 18:36:09.551895 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x6e2a380 con 0x3622840 -7312> 2014-05-19 18:36:09.551926 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7311> 2014-05-19 18:36:09.551939 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x682fc00 con 0x30594a0 -7310> 2014-05-19 18:36:09.552000 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7309> 2014-05-19 18:36:09.552015 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x7999180 con 0x5e7e2c0 -7308> 2014-05-19 18:36:09.552097 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7307> 2014-05-19 18:36:09.552132 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x7733c00 con 0x5e7f1e0 -7306> 2014-05-19 18:36:09.552164 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7305> 2014-05-19 18:36:09.552180 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x63781c0 con 0x5d7bb80 -7304> 2014-05-19 18:36:09.552207 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7303> 2014-05-19 18:36:09.552220 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x5f0ba40 con 0x5e7ec60 -7302> 2014-05-19 18:36:09.552250 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7301> 2014-05-19 18:36:09.552262 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x692d180 con 0x531b760 -7300> 2014-05-19 18:36:09.552292 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7299> 2014-05-19 18:36:09.552308 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x6c4a000 con 0x53bba20 -7298> 2014-05-19 18:36:09.552339 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7297> 2014-05-19 18:36:09.552354 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x7a5ea80 con 0x51cac60 -7296> 2014-05-19 18:36:09.552383 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7295> 2014-05-19 18:36:09.552399 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x60a0c40 con 0x5bd66e0 -7294> 2014-05-19 18:36:09.552429 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7293> 2014-05-19 18:36:09.552443 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 61 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x36f7180 con 0x5bd6840 -7292> 2014-05-19 18:36:09.552473 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7291> 2014-05-19 18:36:09.552485 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x772d340 con 0x3622f20 -7290> 2014-05-19 18:36:09.552512 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7289> 2014-05-19 18:36:09.552527 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:09.551338) v2 ==== 47+0+0 (1225678392 0 0) 0x3511340 con 0x5e7ef20 -7288> 2014-05-19 18:36:09.552556 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7287> 2014-05-19 18:36:09.906897 7fd0456c0700 10 monclient: tick -7286> 2014-05-19 18:36:09.906911 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:35:39.906910) -7285> 2014-05-19 18:36:09.906928 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:36:09.906928; renew after: 2014-05-19 18:38:19.906664) -- no -7284> 2014-05-19 18:36:10.487394 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:13.543984) v2 ==== 47+0+0 (1207734391 0 0) 0x66f8700 con 0x61b0420 -7283> 2014-05-19 18:36:10.487411 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:13.543984) v2 ==== 47+0+0 (1207734391 0 0) 0x784d880 con 0x61ae160 -7282> 2014-05-19 18:36:10.487452 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:13.543984) v2 -- ?+0 0x5e28e00 con 0x61b0420 -7281> 2014-05-19 18:36:10.487485 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7280> 2014-05-19 18:36:10.487502 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:13.543984) v2 -- ?+0 0x78b9500 con 0x61ae160 -7279> 2014-05-19 18:36:10.487535 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7278> 2014-05-19 18:36:10.487574 7fd04f907700 5 osd.0 21 tick -7277> 2014-05-19 18:36:10.487596 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7276> 2014-05-19 18:36:10.487602 7fd04f907700 10 osd.0 21 do_waiters -- start -7275> 2014-05-19 18:36:10.487605 7fd04f907700 10 osd.0 21 do_waiters -- finish -7274> 2014-05-19 18:36:10.487610 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7273> 2014-05-19 18:36:10.651874 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7272> 2014-05-19 18:36:10.651893 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7271> 2014-05-19 18:36:10.651901 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x62e0540 con 0x3622580 -7270> 2014-05-19 18:36:10.651926 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x788a000 con 0x3622840 -7269> 2014-05-19 18:36:10.651964 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x6542fc0 con 0x3622b00 -7268> 2014-05-19 18:36:10.651979 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x6009dc0 con 0x3622dc0 -7267> 2014-05-19 18:36:10.652012 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x682f340 con 0x30594a0 -7266> 2014-05-19 18:36:10.652042 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x3511180 con 0x3622f20 -7265> 2014-05-19 18:36:10.652071 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x51a6540 con 0x5e7ef20 -7264> 2014-05-19 18:36:10.652097 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x76acfc0 con 0x5e7f1e0 -7263> 2014-05-19 18:36:10.652122 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x64afc00 con 0x5e7e2c0 -7262> 2014-05-19 18:36:10.652148 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x6b71dc0 con 0x5d7bb80 -7261> 2014-05-19 18:36:10.652174 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x6c4da40 con 0x5e7ec60 -7260> 2014-05-19 18:36:10.652202 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x692cc40 con 0x531b760 -7259> 2014-05-19 18:36:10.652228 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x7732540 con 0x53bba20 -7258> 2014-05-19 18:36:10.652253 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x36f7880 con 0x51cac60 -7257> 2014-05-19 18:36:10.652237 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x72de380 con 0x3622580 -7256> 2014-05-19 18:36:10.652279 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x60a0540 con 0x5bd66e0 -7255> 2014-05-19 18:36:10.652306 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:10.651899) v2 -- ?+0 0x73c0000 con 0x5bd6840 -7254> 2014-05-19 18:36:10.652351 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7253> 2014-05-19 18:36:10.652374 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x76addc0 con 0x3622b00 -7252> 2014-05-19 18:36:10.652409 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7251> 2014-05-19 18:36:10.652427 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x64ae1c0 con 0x3622dc0 -7250> 2014-05-19 18:36:10.652447 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7249> 2014-05-19 18:36:10.652457 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x653ea80 con 0x3622840 -7248> 2014-05-19 18:36:10.652472 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7247> 2014-05-19 18:36:10.652480 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x5b28380 con 0x3622f20 -7246> 2014-05-19 18:36:10.652498 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7245> 2014-05-19 18:36:10.652509 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x7998fc0 con 0x5e7e2c0 -7244> 2014-05-19 18:36:10.652552 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7243> 2014-05-19 18:36:10.652568 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x682fdc0 con 0x30594a0 -7242> 2014-05-19 18:36:10.652632 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7241> 2014-05-19 18:36:10.652699 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x6d616c0 con 0x5e7ef20 -7240> 2014-05-19 18:36:10.652737 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7239> 2014-05-19 18:36:10.652753 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x6378e00 con 0x5d7bb80 -7238> 2014-05-19 18:36:10.652785 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7237> 2014-05-19 18:36:10.652794 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x5f0a380 con 0x5e7ec60 -7236> 2014-05-19 18:36:10.652812 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7235> 2014-05-19 18:36:10.652824 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x692c000 con 0x531b760 -7234> 2014-05-19 18:36:10.652841 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7233> 2014-05-19 18:36:10.652849 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x55f48c0 con 0x53bba20 -7232> 2014-05-19 18:36:10.652866 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7231> 2014-05-19 18:36:10.652876 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x7a5f180 con 0x51cac60 -7230> 2014-05-19 18:36:10.652893 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7229> 2014-05-19 18:36:10.652901 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x60a1180 con 0x5bd66e0 -7228> 2014-05-19 18:36:10.652918 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7227> 2014-05-19 18:36:10.652926 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 62 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x36f7500 con 0x5bd6840 -7226> 2014-05-19 18:36:10.652940 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7225> 2014-05-19 18:36:10.652948 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:10.651899) v2 ==== 47+0+0 (2581966450 0 0) 0x77321c0 con 0x5e7f1e0 -7224> 2014-05-19 18:36:10.652964 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7223> 2014-05-19 18:36:10.773351 7fd04e905700 20 KeyValueStore::op_tp worker waiting -7222> 2014-05-19 18:36:10.773454 7fd043ebd700 20 OSD::op_tp worker waiting -7221> 2014-05-19 18:36:10.945680 7fd0396a8700 20 OSD::command_tp worker waiting -7220> 2014-05-19 18:36:11.084554 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:16.169917) v2 ==== 47+0+0 (4157372063 0 0) 0x5dffdc0 con 0x5b21600 -7219> 2014-05-19 18:36:11.084624 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:16.169917) v2 ==== 47+0+0 (4157372063 0 0) 0x70cda40 con 0x5f5b760 -7218> 2014-05-19 18:36:11.084629 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:16.169917) v2 -- ?+0 0x784d880 con 0x5b21600 -7217> 2014-05-19 18:36:11.084694 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7216> 2014-05-19 18:36:11.084727 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:16.169917) v2 -- ?+0 0x66f8700 con 0x5f5b760 -7215> 2014-05-19 18:36:11.084780 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7214> 2014-05-19 18:36:11.317504 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:14.374132) v2 ==== 47+0+0 (693084978 0 0) 0x6aa5c00 con 0x618fe40 -7213> 2014-05-19 18:36:11.317511 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:14.374132) v2 ==== 47+0+0 (693084978 0 0) 0x5bea540 con 0x61ae420 -7212> 2014-05-19 18:36:11.317551 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:14.374132) v2 -- ?+0 0x70cda40 con 0x618fe40 -7211> 2014-05-19 18:36:11.317576 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7210> 2014-05-19 18:36:11.317593 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:14.374132) v2 -- ?+0 0x5dffdc0 con 0x61ae420 -7209> 2014-05-19 18:36:11.317632 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7208> 2014-05-19 18:36:11.339929 7fd03c6ae700 20 OSD::op_tp worker waiting -7207> 2014-05-19 18:36:11.339955 7fd041eb9700 20 OSD::op_tp worker waiting -7206> 2014-05-19 18:36:11.340031 7fd03e6b2700 20 OSD::op_tp worker waiting -7205> 2014-05-19 18:36:11.340440 7fd03d6b0700 20 OSD::op_tp worker waiting -7204> 2014-05-19 18:36:11.340620 7fd0436bc700 20 OSD::op_tp worker waiting -7203> 2014-05-19 18:36:11.340742 7fd03bead700 20 OSD::op_tp worker waiting -7202> 2014-05-19 18:36:11.340761 7fd03f6b4700 20 OSD::op_tp worker waiting -7201> 2014-05-19 18:36:11.354576 7fd0446be700 20 OSD::op_tp worker waiting -7200> 2014-05-19 18:36:11.354603 7fd03feb5700 20 OSD::op_tp worker waiting -7199> 2014-05-19 18:36:11.361593 7fd040eb7700 20 OSD::op_tp worker waiting -7198> 2014-05-19 18:36:11.366846 7fd0406b6700 20 OSD::op_tp worker waiting -7197> 2014-05-19 18:36:11.369445 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:16.454804) v2 ==== 47+0+0 (407984985 0 0) 0x60f6e00 con 0x31e1600 -7196> 2014-05-19 18:36:11.369496 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:16.454804) v2 -- ?+0 0x5bea540 con 0x31e1600 -7195> 2014-05-19 18:36:11.369498 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:16.454804) v2 ==== 47+0+0 (407984985 0 0) 0x75b0e00 con 0x61b0160 -7194> 2014-05-19 18:36:11.369584 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7193> 2014-05-19 18:36:11.369618 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:16.454804) v2 -- ?+0 0x6aa5c00 con 0x61b0160 -7192> 2014-05-19 18:36:11.369649 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7191> 2014-05-19 18:36:11.388454 7fd039ea9700 20 OSD::disk_tp worker waiting -7190> 2014-05-19 18:36:11.392042 7fd03aeab700 20 OSD::op_tp worker waiting -7189> 2014-05-19 18:36:11.392061 7fd03eeb3700 20 OSD::op_tp worker waiting -7188> 2014-05-19 18:36:11.409949 7fd042ebb700 20 OSD::op_tp worker waiting -7187> 2014-05-19 18:36:11.409975 7fd03deb1700 20 OSD::op_tp worker waiting -7186> 2014-05-19 18:36:11.418497 7fd04e104700 20 KeyValueStore::op_tp worker waiting -7185> 2014-05-19 18:36:11.418504 7fd03b6ac700 20 OSD::op_tp worker waiting -7184> 2014-05-19 18:36:11.418525 7fd0426ba700 20 OSD::op_tp worker waiting -7183> 2014-05-19 18:36:11.418545 7fd0416b8700 20 OSD::op_tp worker waiting -7182> 2014-05-19 18:36:11.418616 7fd03ceaf700 20 OSD::op_tp worker waiting -7181> 2014-05-19 18:36:11.466943 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:11.466752) v2 ==== 47+0+0 (1778339182 0 0) 0x70a8700 con 0x4cb8160 -7180> 2014-05-19 18:36:11.466994 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:11.466752) v2 -- ?+0 0x60f6e00 con 0x4cb8160 -7179> 2014-05-19 18:36:11.467002 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:11.466752) v2 ==== 47+0+0 (1778339182 0 0) 0x5debc00 con 0x4cb82c0 -7178> 2014-05-19 18:36:11.467096 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7177> 2014-05-19 18:36:11.467240 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:11.466752) v2 -- ?+0 0x75b0e00 con 0x4cb82c0 -7176> 2014-05-19 18:36:11.467304 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7175> 2014-05-19 18:36:11.487695 7fd04f907700 5 osd.0 21 tick -7174> 2014-05-19 18:36:11.487717 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7173> 2014-05-19 18:36:11.487722 7fd04f907700 10 osd.0 21 do_waiters -- start -7172> 2014-05-19 18:36:11.487724 7fd04f907700 10 osd.0 21 do_waiters -- finish -7171> 2014-05-19 18:36:11.487764 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7170> 2014-05-19 18:36:12.009538 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:17.094920) v2 ==== 47+0+0 (2844132828 0 0) 0x8388000 con 0x3094160 -7169> 2014-05-19 18:36:12.009552 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 58 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:17.094920) v2 ==== 47+0+0 (2844132828 0 0) 0x73c16c0 con 0x30946e0 -7168> 2014-05-19 18:36:12.009588 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:17.094920) v2 -- ?+0 0x5debc00 con 0x3094160 -7167> 2014-05-19 18:36:12.009679 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7166> 2014-05-19 18:36:12.009708 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:17.094920) v2 -- ?+0 0x70a8700 con 0x30946e0 -7165> 2014-05-19 18:36:12.009756 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7164> 2014-05-19 18:36:12.418116 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:15.474741) v2 ==== 47+0+0 (1604676225 0 0) 0x66f8380 con 0x618fe40 -7163> 2014-05-19 18:36:12.418124 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:15.474741) v2 ==== 47+0+0 (1604676225 0 0) 0x55188c0 con 0x61ae420 -7162> 2014-05-19 18:36:12.418175 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.474741) v2 -- ?+0 0x8388000 con 0x618fe40 -7161> 2014-05-19 18:36:12.418249 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7160> 2014-05-19 18:36:12.418288 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.474741) v2 -- ?+0 0x73c16c0 con 0x61ae420 -7159> 2014-05-19 18:36:12.418328 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7158> 2014-05-19 18:36:12.487808 7fd04f907700 5 osd.0 21 tick -7157> 2014-05-19 18:36:12.487826 7fd04f907700 7 osd.0 21 do_mon_report -7156> 2014-05-19 18:36:12.487831 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -7155> 2014-05-19 18:36:12.487837 7fd04f907700 20 osd.0 21 send_pg_stats -7154> 2014-05-19 18:36:12.487849 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7153> 2014-05-19 18:36:12.487854 7fd04f907700 10 osd.0 21 do_waiters -- start -7152> 2014-05-19 18:36:12.487858 7fd04f907700 10 osd.0 21 do_waiters -- finish -7151> 2014-05-19 18:36:12.487886 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7150> 2014-05-19 18:36:12.567456 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:12.567260) v2 ==== 47+0+0 (2231351092 0 0) 0x662ba40 con 0x4cb82c0 -7149> 2014-05-19 18:36:12.567523 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.567260) v2 -- ?+0 0x66f8380 con 0x4cb82c0 -7148> 2014-05-19 18:36:12.567562 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7147> 2014-05-19 18:36:12.567478 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:12.567260) v2 ==== 47+0+0 (2231351092 0 0) 0x70a88c0 con 0x4cb8160 -7146> 2014-05-19 18:36:12.567623 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.567260) v2 -- ?+0 0x55188c0 con 0x4cb8160 -7145> 2014-05-19 18:36:12.567669 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7144> 2014-05-19 18:36:12.773447 7fd04e905700 20 KeyValueStore::op_tp worker waiting -7143> 2014-05-19 18:36:12.773535 7fd043ebd700 20 OSD::op_tp worker waiting -7142> 2014-05-19 18:36:12.784999 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:17.870423) v2 ==== 47+0+0 (1366916650 0 0) 0x5497340 con 0x5f5b760 -7141> 2014-05-19 18:36:12.785049 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:17.870423) v2 -- ?+0 0x662ba40 con 0x5f5b760 -7140> 2014-05-19 18:36:12.785083 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7139> 2014-05-19 18:36:12.784981 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:17.870423) v2 ==== 47+0+0 (1366916650 0 0) 0x2ee1500 con 0x5b21600 -7138> 2014-05-19 18:36:12.785133 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:17.870423) v2 -- ?+0 0x70a88c0 con 0x5b21600 -7137> 2014-05-19 18:36:12.785168 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7136> 2014-05-19 18:36:12.945767 7fd0396a8700 20 OSD::command_tp worker waiting -7135> 2014-05-19 18:36:12.952430 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7134> 2014-05-19 18:36:12.952449 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7133> 2014-05-19 18:36:12.952457 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x3438a80 con 0x3622580 -7132> 2014-05-19 18:36:12.952483 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x5519880 con 0x3622840 -7131> 2014-05-19 18:36:12.952521 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x6efefc0 con 0x3622b00 -7130> 2014-05-19 18:36:12.952536 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x60096c0 con 0x3622dc0 -7129> 2014-05-19 18:36:12.952569 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x7998a80 con 0x30594a0 -7128> 2014-05-19 18:36:12.952616 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x3511500 con 0x3622f20 -7127> 2014-05-19 18:36:12.952652 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x76adc00 con 0x5e7ef20 -7126> 2014-05-19 18:36:12.952678 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x6b70380 con 0x5e7f1e0 -7125> 2014-05-19 18:36:12.952703 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x64ae000 con 0x5e7e2c0 -7124> 2014-05-19 18:36:12.952725 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x51a6a80 con 0x5d7bb80 -7123> 2014-05-19 18:36:12.952750 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x6113880 con 0x5e7ec60 -7122> 2014-05-19 18:36:12.952775 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x6c4c540 con 0x531b760 -7121> 2014-05-19 18:36:12.952802 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x71221c0 con 0x53bba20 -7120> 2014-05-19 18:36:12.952828 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x60a16c0 con 0x51cac60 -7119> 2014-05-19 18:36:12.952821 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x72dfdc0 con 0x3622580 -7118> 2014-05-19 18:36:12.952855 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x77e81c0 con 0x5bd66e0 -7117> 2014-05-19 18:36:12.952880 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:12.952455) v2 -- ?+0 0x3438380 con 0x5bd6840 -7116> 2014-05-19 18:36:12.952926 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7115> 2014-05-19 18:36:12.952949 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x76ac1c0 con 0x3622b00 -7114> 2014-05-19 18:36:12.952980 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7113> 2014-05-19 18:36:12.952997 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x64afdc0 con 0x3622dc0 -7112> 2014-05-19 18:36:12.953026 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7111> 2014-05-19 18:36:12.953157 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x2f4efc0 con 0x3622f20 -7110> 2014-05-19 18:36:12.953193 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7109> 2014-05-19 18:36:12.953202 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x7178c40 con 0x3622840 -7108> 2014-05-19 18:36:12.953219 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7107> 2014-05-19 18:36:12.953227 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x6d61180 con 0x5e7ef20 -7106> 2014-05-19 18:36:12.953243 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7105> 2014-05-19 18:36:12.953248 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x7733a40 con 0x5e7f1e0 -7104> 2014-05-19 18:36:12.953262 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7103> 2014-05-19 18:36:12.953311 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x7998700 con 0x5e7e2c0 -7102> 2014-05-19 18:36:12.953352 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7101> 2014-05-19 18:36:12.953369 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x6710fc0 con 0x5d7bb80 -7100> 2014-05-19 18:36:12.953399 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -7099> 2014-05-19 18:36:12.953409 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x5f0b180 con 0x5e7ec60 -7098> 2014-05-19 18:36:12.953424 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7097> 2014-05-19 18:36:12.953433 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x692dc00 con 0x531b760 -7096> 2014-05-19 18:36:12.953450 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -7095> 2014-05-19 18:36:12.953458 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x55f4380 con 0x53bba20 -7094> 2014-05-19 18:36:12.953473 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7093> 2014-05-19 18:36:12.953482 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x7a5f340 con 0x51cac60 -7092> 2014-05-19 18:36:12.953499 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7091> 2014-05-19 18:36:12.953507 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x60a0fc0 con 0x5bd66e0 -7090> 2014-05-19 18:36:12.953525 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7089> 2014-05-19 18:36:12.953533 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 63 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x36f6700 con 0x5bd6840 -7088> 2014-05-19 18:36:12.953548 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -7087> 2014-05-19 18:36:12.953555 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:12.952455) v2 ==== 47+0+0 (2428470642 0 0) 0x682e540 con 0x30594a0 -7086> 2014-05-19 18:36:12.953570 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7085> 2014-05-19 18:36:13.340012 7fd03c6ae700 20 OSD::op_tp worker waiting -7084> 2014-05-19 18:36:13.340030 7fd041eb9700 20 OSD::op_tp worker waiting -7083> 2014-05-19 18:36:13.340102 7fd03e6b2700 20 OSD::op_tp worker waiting -7082> 2014-05-19 18:36:13.340510 7fd03d6b0700 20 OSD::op_tp worker waiting -7081> 2014-05-19 18:36:13.340695 7fd0436bc700 20 OSD::op_tp worker waiting -7080> 2014-05-19 18:36:13.340818 7fd03bead700 20 OSD::op_tp worker waiting -7079> 2014-05-19 18:36:13.340847 7fd03f6b4700 20 OSD::op_tp worker waiting -7078> 2014-05-19 18:36:13.354675 7fd0446be700 20 OSD::op_tp worker waiting -7077> 2014-05-19 18:36:13.354699 7fd03feb5700 20 OSD::op_tp worker waiting -7076> 2014-05-19 18:36:13.361692 7fd040eb7700 20 OSD::op_tp worker waiting -7075> 2014-05-19 18:36:13.366944 7fd0406b6700 20 OSD::op_tp worker waiting -7074> 2014-05-19 18:36:13.381789 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:13.381605) v2 ==== 47+0+0 (262624680 0 0) 0x6113dc0 con 0x4cb8420 -7073> 2014-05-19 18:36:13.381848 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:13.381605) v2 -- ?+0 0x5497340 con 0x4cb8420 -7072> 2014-05-19 18:36:13.381878 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7071> 2014-05-19 18:36:13.381789 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:13.381605) v2 ==== 47+0+0 (262624680 0 0) 0x70a9a40 con 0x4cb86e0 -7070> 2014-05-19 18:36:13.381902 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:13.381605) v2 -- ?+0 0x2ee1500 con 0x4cb86e0 -7069> 2014-05-19 18:36:13.381944 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7068> 2014-05-19 18:36:13.388536 7fd039ea9700 20 OSD::disk_tp worker waiting -7067> 2014-05-19 18:36:13.392124 7fd03aeab700 20 OSD::op_tp worker waiting -7066> 2014-05-19 18:36:13.392147 7fd03eeb3700 20 OSD::op_tp worker waiting -7065> 2014-05-19 18:36:13.410048 7fd042ebb700 20 OSD::op_tp worker waiting -7064> 2014-05-19 18:36:13.410073 7fd03deb1700 20 OSD::op_tp worker waiting -7063> 2014-05-19 18:36:13.418594 7fd04e104700 20 KeyValueStore::op_tp worker waiting -7062> 2014-05-19 18:36:13.418601 7fd03b6ac700 20 OSD::op_tp worker waiting -7061> 2014-05-19 18:36:13.418623 7fd0416b8700 20 OSD::op_tp worker waiting -7060> 2014-05-19 18:36:13.418641 7fd0426ba700 20 OSD::op_tp worker waiting -7059> 2014-05-19 18:36:13.418681 7fd03ceaf700 20 OSD::op_tp worker waiting -7058> 2014-05-19 18:36:13.487947 7fd04f907700 5 osd.0 21 tick -7057> 2014-05-19 18:36:13.487968 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7056> 2014-05-19 18:36:13.487973 7fd04f907700 10 osd.0 21 do_waiters -- start -7055> 2014-05-19 18:36:13.487975 7fd04f907700 10 osd.0 21 do_waiters -- finish -7054> 2014-05-19 18:36:13.488016 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7053> 2014-05-19 18:36:13.709944 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:18.795392) v2 ==== 47+0+0 (7765411 0 0) 0x66f9dc0 con 0x30946e0 -7052> 2014-05-19 18:36:13.709966 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 59 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:18.795392) v2 ==== 47+0+0 (7765411 0 0) 0x5c04e00 con 0x3094160 -7051> 2014-05-19 18:36:13.709991 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:18.795392) v2 -- ?+0 0x70a9a40 con 0x30946e0 -7050> 2014-05-19 18:36:13.710029 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7049> 2014-05-19 18:36:13.710056 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:18.795392) v2 -- ?+0 0x6113dc0 con 0x3094160 -7048> 2014-05-19 18:36:13.710097 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -7047> 2014-05-19 18:36:14.456247 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:17.512941) v2 ==== 47+0+0 (607953872 0 0) 0x5e28380 con 0x4cb9340 -7046> 2014-05-19 18:36:14.456277 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:17.512941) v2 ==== 47+0+0 (607953872 0 0) 0x79e6fc0 con 0x4cb8dc0 -7045> 2014-05-19 18:36:14.456330 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:17.512941) v2 -- ?+0 0x5c04e00 con 0x4cb9340 -7044> 2014-05-19 18:36:14.456368 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7043> 2014-05-19 18:36:14.456388 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:17.512941) v2 -- ?+0 0x66f9dc0 con 0x4cb8dc0 -7042> 2014-05-19 18:36:14.456426 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -7041> 2014-05-19 18:36:14.482324 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:14.482153) v2 ==== 47+0+0 (2475724377 0 0) 0x72e4c40 con 0x4cb86e0 -7040> 2014-05-19 18:36:14.482371 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:14.482153) v2 -- ?+0 0x79e6fc0 con 0x4cb86e0 -7039> 2014-05-19 18:36:14.482402 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7038> 2014-05-19 18:36:14.482381 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:14.482153) v2 ==== 47+0+0 (2475724377 0 0) 0x6112e00 con 0x4cb8420 -7037> 2014-05-19 18:36:14.482432 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:14.482153) v2 -- ?+0 0x5e28380 con 0x4cb8420 -7036> 2014-05-19 18:36:14.482463 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -7035> 2014-05-19 18:36:14.488046 7fd04f907700 5 osd.0 21 tick -7034> 2014-05-19 18:36:14.488067 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -7033> 2014-05-19 18:36:14.488072 7fd04f907700 10 osd.0 21 do_waiters -- start -7032> 2014-05-19 18:36:14.488074 7fd04f907700 10 osd.0 21 do_waiters -- finish -7031> 2014-05-19 18:36:14.488115 7fd03a6aa700 20 OSD::recovery_tp worker waiting -7030> 2014-05-19 18:36:14.718613 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:17.775285) v2 ==== 47+0+0 (3447027860 0 0) 0x5519500 con 0x61ae420 -7029> 2014-05-19 18:36:14.718659 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:17.775285) v2 -- ?+0 0x72e4c40 con 0x61ae420 -7028> 2014-05-19 18:36:14.718705 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7027> 2014-05-19 18:36:14.718648 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:17.775285) v2 ==== 47+0+0 (3447027860 0 0) 0x66f9500 con 0x618fe40 -7026> 2014-05-19 18:36:14.718757 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:17.775285) v2 -- ?+0 0x6112e00 con 0x618fe40 -7025> 2014-05-19 18:36:14.718801 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -7024> 2014-05-19 18:36:14.773537 7fd04e905700 20 KeyValueStore::op_tp worker waiting -7023> 2014-05-19 18:36:14.773617 7fd043ebd700 20 OSD::op_tp worker waiting -7022> 2014-05-19 18:36:14.868000 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:14.867829) v2 ==== 47+0+0 (1143273614 0 0) 0x70a8e00 con 0x4cb8160 -7021> 2014-05-19 18:36:14.868048 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:14.867829) v2 -- ?+0 0x5519500 con 0x4cb8160 -7020> 2014-05-19 18:36:14.868095 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7019> 2014-05-19 18:36:14.868046 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:14.867829) v2 ==== 47+0+0 (1143273614 0 0) 0x662b880 con 0x4cb82c0 -7018> 2014-05-19 18:36:14.868160 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:14.867829) v2 -- ?+0 0x66f9500 con 0x4cb82c0 -7017> 2014-05-19 18:36:14.868200 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -7016> 2014-05-19 18:36:14.945862 7fd0396a8700 20 OSD::command_tp worker waiting -7015> 2014-05-19 18:36:15.253018 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7014> 2014-05-19 18:36:15.253037 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -7013> 2014-05-19 18:36:15.253048 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x692da40 con 0x3622580 -7012> 2014-05-19 18:36:15.253081 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x6542c40 con 0x3622840 -7011> 2014-05-19 18:36:15.253105 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x6009500 con 0x3622b00 -7010> 2014-05-19 18:36:15.253133 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x7999880 con 0x3622dc0 -7009> 2014-05-19 18:36:15.253161 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x7998fc0 con 0x30594a0 -7008> 2014-05-19 18:36:15.253187 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x5b28380 con 0x3622f20 -7007> 2014-05-19 18:36:15.253213 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x653ea80 con 0x5e7ef20 -7006> 2014-05-19 18:36:15.253246 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x64ae1c0 con 0x5e7f1e0 -7005> 2014-05-19 18:36:15.253271 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x76addc0 con 0x5e7e2c0 -7004> 2014-05-19 18:36:15.253295 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x72de380 con 0x5d7bb80 -7003> 2014-05-19 18:36:15.253319 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x3511340 con 0x5e7ec60 -7002> 2014-05-19 18:36:15.253342 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x772d340 con 0x531b760 -7001> 2014-05-19 18:36:15.253367 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x36f7180 con 0x53bba20 -7000> 2014-05-19 18:36:15.253393 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x60a0c40 con 0x51cac60 -6999> 2014-05-19 18:36:15.253419 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x7a5ea80 con 0x5bd66e0 -6998> 2014-05-19 18:36:15.253424 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x2f4fdc0 con 0x3622840 -6997> 2014-05-19 18:36:15.253444 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:15.253047) v2 -- ?+0 0x6c4a000 con 0x5bd6840 -6996> 2014-05-19 18:36:15.253494 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6995> 2014-05-19 18:36:15.253536 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x64aec40 con 0x3622dc0 -6994> 2014-05-19 18:36:15.253561 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6993> 2014-05-19 18:36:15.253587 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x6fe7a40 con 0x3622580 -6992> 2014-05-19 18:36:15.253647 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6991> 2014-05-19 18:36:15.253667 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x711e700 con 0x5e7e2c0 -6990> 2014-05-19 18:36:15.253686 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6989> 2014-05-19 18:36:15.253707 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x6b71500 con 0x3622f20 -6988> 2014-05-19 18:36:15.253746 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6987> 2014-05-19 18:36:15.253764 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x83881c0 con 0x5e7ef20 -6986> 2014-05-19 18:36:15.253800 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6985> 2014-05-19 18:36:15.253841 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x7733340 con 0x5e7f1e0 -6984> 2014-05-19 18:36:15.253866 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6983> 2014-05-19 18:36:15.253873 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x6711180 con 0x5d7bb80 -6982> 2014-05-19 18:36:15.253888 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6981> 2014-05-19 18:36:15.253893 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x5f06000 con 0x5e7ec60 -6980> 2014-05-19 18:36:15.253945 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6979> 2014-05-19 18:36:15.253954 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x6542000 con 0x531b760 -6978> 2014-05-19 18:36:15.253974 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6977> 2014-05-19 18:36:15.253986 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x55f4fc0 con 0x53bba20 -6976> 2014-05-19 18:36:15.254003 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6975> 2014-05-19 18:36:15.254012 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x7a5e1c0 con 0x51cac60 -6974> 2014-05-19 18:36:15.254026 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6973> 2014-05-19 18:36:15.254033 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x7122700 con 0x5bd66e0 -6972> 2014-05-19 18:36:15.254050 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6971> 2014-05-19 18:36:15.254058 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 64 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x36f7340 con 0x5bd6840 -6970> 2014-05-19 18:36:15.254073 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6969> 2014-05-19 18:36:15.254080 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x6009c00 con 0x30594a0 -6968> 2014-05-19 18:36:15.254095 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6967> 2014-05-19 18:36:15.254102 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:15.253047) v2 ==== 47+0+0 (3695278057 0 0) 0x5496000 con 0x3622b00 -6966> 2014-05-19 18:36:15.254117 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6965> 2014-05-19 18:36:15.340095 7fd03c6ae700 20 OSD::op_tp worker waiting -6964> 2014-05-19 18:36:15.340119 7fd041eb9700 20 OSD::op_tp worker waiting -6963> 2014-05-19 18:36:15.340206 7fd03e6b2700 20 OSD::op_tp worker waiting -6962> 2014-05-19 18:36:15.340588 7fd03d6b0700 20 OSD::op_tp worker waiting -6961> 2014-05-19 18:36:15.340767 7fd0436bc700 20 OSD::op_tp worker waiting -6960> 2014-05-19 18:36:15.340891 7fd03bead700 20 OSD::op_tp worker waiting -6959> 2014-05-19 18:36:15.340944 7fd03f6b4700 20 OSD::op_tp worker waiting -6958> 2014-05-19 18:36:15.354775 7fd0446be700 20 OSD::op_tp worker waiting -6957> 2014-05-19 18:36:15.354831 7fd03feb5700 20 OSD::op_tp worker waiting -6956> 2014-05-19 18:36:15.361789 7fd040eb7700 20 OSD::op_tp worker waiting -6955> 2014-05-19 18:36:15.367041 7fd0406b6700 20 OSD::op_tp worker waiting -6954> 2014-05-19 18:36:15.388617 7fd039ea9700 20 OSD::disk_tp worker waiting -6953> 2014-05-19 18:36:15.392207 7fd03aeab700 20 OSD::op_tp worker waiting -6952> 2014-05-19 18:36:15.392226 7fd03eeb3700 20 OSD::op_tp worker waiting -6951> 2014-05-19 18:36:15.410148 7fd042ebb700 20 OSD::op_tp worker waiting -6950> 2014-05-19 18:36:15.410171 7fd03deb1700 20 OSD::op_tp worker waiting -6949> 2014-05-19 18:36:15.418689 7fd04e104700 20 KeyValueStore::op_tp worker waiting -6948> 2014-05-19 18:36:15.418691 7fd03b6ac700 20 OSD::op_tp worker waiting -6947> 2014-05-19 18:36:15.418730 7fd0416b8700 20 OSD::op_tp worker waiting -6946> 2014-05-19 18:36:15.418750 7fd0426ba700 20 OSD::op_tp worker waiting -6945> 2014-05-19 18:36:15.418767 7fd03ceaf700 20 OSD::op_tp worker waiting -6944> 2014-05-19 18:36:15.488148 7fd04f907700 5 osd.0 21 tick -6943> 2014-05-19 18:36:15.488168 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -6942> 2014-05-19 18:36:15.488173 7fd04f907700 10 osd.0 21 do_waiters -- start -6941> 2014-05-19 18:36:15.488175 7fd04f907700 10 osd.0 21 do_waiters -- finish -6940> 2014-05-19 18:36:15.488217 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6939> 2014-05-19 18:36:15.556820 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:18.613504) v2 ==== 47+0+0 (1482306936 0 0) 0x5e28c40 con 0x4cb9340 -6938> 2014-05-19 18:36:15.556871 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:18.613504) v2 -- ?+0 0x662b880 con 0x4cb9340 -6937> 2014-05-19 18:36:15.556908 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6936> 2014-05-19 18:36:15.556830 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:18.613504) v2 ==== 47+0+0 (1482306936 0 0) 0x79e6540 con 0x4cb8dc0 -6935> 2014-05-19 18:36:15.556981 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:18.613504) v2 -- ?+0 0x70a8e00 con 0x4cb8dc0 -6934> 2014-05-19 18:36:15.557031 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6933> 2014-05-19 18:36:16.387817 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:19.444525) v2 ==== 47+0+0 (127297417 0 0) 0x5c46380 con 0x61ae160 -6932> 2014-05-19 18:36:16.387859 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:19.444525) v2 -- ?+0 0x79e6540 con 0x61ae160 -6931> 2014-05-19 18:36:16.387870 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:19.444525) v2 ==== 47+0+0 (127297417 0 0) 0x66f81c0 con 0x61b0420 -6930> 2014-05-19 18:36:16.387936 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6929> 2014-05-19 18:36:16.387988 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:19.444525) v2 -- ?+0 0x5e28c40 con 0x61b0420 -6928> 2014-05-19 18:36:16.388029 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6927> 2014-05-19 18:36:16.488242 7fd04f907700 5 osd.0 21 tick -6926> 2014-05-19 18:36:16.488301 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6925> 2014-05-19 18:36:16.488298 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.09 < max 0.5 = yes -6924> 2014-05-19 18:36:16.488330 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -6923> 2014-05-19 18:36:16.488336 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 219.212 < min (86400 seconds) -6922> 2014-05-19 18:36:16.488360 7fd04f907700 20 osd.0 21 sched_scrub done -6921> 2014-05-19 18:36:16.488366 7fd04f907700 10 osd.0 21 do_waiters -- start -6920> 2014-05-19 18:36:16.488370 7fd04f907700 10 osd.0 21 do_waiters -- finish -6919> 2014-05-19 18:36:16.773630 7fd04e905700 20 KeyValueStore::op_tp worker waiting -6918> 2014-05-19 18:36:16.773703 7fd043ebd700 20 OSD::op_tp worker waiting -6917> 2014-05-19 18:36:16.782884 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:16.782704) v2 ==== 47+0+0 (2157199842 0 0) 0x5cbcfc0 con 0x4cb86e0 -6916> 2014-05-19 18:36:16.782936 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:16.782704) v2 -- ?+0 0x5c46380 con 0x4cb86e0 -6915> 2014-05-19 18:36:16.782939 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:16.782704) v2 ==== 47+0+0 (2157199842 0 0) 0x5fc3880 con 0x4cb8420 -6914> 2014-05-19 18:36:16.783027 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6913> 2014-05-19 18:36:16.783056 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:16.782704) v2 -- ?+0 0x66f81c0 con 0x4cb8420 -6912> 2014-05-19 18:36:16.783103 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6911> 2014-05-19 18:36:16.885420 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:21.970980) v2 ==== 47+0+0 (12547711 0 0) 0x5497a40 con 0x5f5b760 -6910> 2014-05-19 18:36:16.885419 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:21.970980) v2 ==== 47+0+0 (12547711 0 0) 0x2ee1dc0 con 0x5b21600 -6909> 2014-05-19 18:36:16.885480 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.970980) v2 -- ?+0 0x5fc3880 con 0x5f5b760 -6908> 2014-05-19 18:36:16.885515 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6907> 2014-05-19 18:36:16.885535 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.970980) v2 -- ?+0 0x5cbcfc0 con 0x5b21600 -6906> 2014-05-19 18:36:16.885622 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6905> 2014-05-19 18:36:16.945958 7fd0396a8700 20 OSD::command_tp worker waiting -6904> 2014-05-19 18:36:17.019172 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:20.075885) v2 ==== 47+0+0 (2894094234 0 0) 0x77e8380 con 0x61ae420 -6903> 2014-05-19 18:36:17.019202 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:20.075885) v2 ==== 47+0+0 (2894094234 0 0) 0x6b71a40 con 0x618fe40 -6902> 2014-05-19 18:36:17.019221 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:20.075885) v2 -- ?+0 0x2ee1dc0 con 0x61ae420 -6901> 2014-05-19 18:36:17.019255 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6900> 2014-05-19 18:36:17.019311 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:20.075885) v2 -- ?+0 0x5497a40 con 0x618fe40 -6899> 2014-05-19 18:36:17.019349 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6898> 2014-05-19 18:36:17.168629 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:17.168430) v2 ==== 47+0+0 (3133569738 0 0) 0x70a8540 con 0x4cb8160 -6897> 2014-05-19 18:36:17.168670 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:17.168430) v2 -- ?+0 0x77e8380 con 0x4cb8160 -6896> 2014-05-19 18:36:17.168706 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6895> 2014-05-19 18:36:17.168725 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:17.168430) v2 ==== 47+0+0 (3133569738 0 0) 0x662bdc0 con 0x4cb82c0 -6894> 2014-05-19 18:36:17.168762 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:17.168430) v2 -- ?+0 0x6b71a40 con 0x4cb82c0 -6893> 2014-05-19 18:36:17.168796 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6892> 2014-05-19 18:36:17.269686 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:22.355268) v2 ==== 47+0+0 (2329504851 0 0) 0x5beba40 con 0x61b0160 -6891> 2014-05-19 18:36:17.269686 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:22.355268) v2 ==== 47+0+0 (2329504851 0 0) 0x7a36fc0 con 0x31e1600 -6890> 2014-05-19 18:36:17.269725 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:22.355268) v2 -- ?+0 0x662bdc0 con 0x61b0160 -6889> 2014-05-19 18:36:17.269757 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6888> 2014-05-19 18:36:17.269804 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:22.355268) v2 -- ?+0 0x70a8540 con 0x31e1600 -6887> 2014-05-19 18:36:17.269859 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6886> 2014-05-19 18:36:17.340173 7fd03c6ae700 20 OSD::op_tp worker waiting -6885> 2014-05-19 18:36:17.340194 7fd041eb9700 20 OSD::op_tp worker waiting -6884> 2014-05-19 18:36:17.340286 7fd03e6b2700 20 OSD::op_tp worker waiting -6883> 2014-05-19 18:36:17.340670 7fd03d6b0700 20 OSD::op_tp worker waiting -6882> 2014-05-19 18:36:17.340837 7fd0436bc700 20 OSD::op_tp worker waiting -6881> 2014-05-19 18:36:17.340962 7fd03bead700 20 OSD::op_tp worker waiting -6880> 2014-05-19 18:36:17.341028 7fd03f6b4700 20 OSD::op_tp worker waiting -6879> 2014-05-19 18:36:17.354873 7fd0446be700 20 OSD::op_tp worker waiting -6878> 2014-05-19 18:36:17.354916 7fd03feb5700 20 OSD::op_tp worker waiting -6877> 2014-05-19 18:36:17.361887 7fd040eb7700 20 OSD::op_tp worker waiting -6876> 2014-05-19 18:36:17.367139 7fd0406b6700 20 OSD::op_tp worker waiting -6875> 2014-05-19 18:36:17.388689 7fd039ea9700 20 OSD::disk_tp worker waiting -6874> 2014-05-19 18:36:17.392288 7fd03aeab700 20 OSD::op_tp worker waiting -6873> 2014-05-19 18:36:17.392309 7fd03eeb3700 20 OSD::op_tp worker waiting -6872> 2014-05-19 18:36:17.410247 7fd042ebb700 20 OSD::op_tp worker waiting -6871> 2014-05-19 18:36:17.410273 7fd03deb1700 20 OSD::op_tp worker waiting -6870> 2014-05-19 18:36:17.418798 7fd04e104700 20 KeyValueStore::op_tp worker waiting -6869> 2014-05-19 18:36:17.418803 7fd03b6ac700 20 OSD::op_tp worker waiting -6868> 2014-05-19 18:36:17.418823 7fd0416b8700 20 OSD::op_tp worker waiting -6867> 2014-05-19 18:36:17.418848 7fd0426ba700 20 OSD::op_tp worker waiting -6866> 2014-05-19 18:36:17.418894 7fd03ceaf700 20 OSD::op_tp worker waiting -6865> 2014-05-19 18:36:17.488271 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:20.545016) v2 ==== 47+0+0 (3102559781 0 0) 0x5c468c0 con 0x61ae160 -6864> 2014-05-19 18:36:17.488322 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:20.545016) v2 -- ?+0 0x7a36fc0 con 0x61ae160 -6863> 2014-05-19 18:36:17.488335 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:20.545016) v2 ==== 47+0+0 (3102559781 0 0) 0x66f8540 con 0x61b0420 -6862> 2014-05-19 18:36:17.488368 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6861> 2014-05-19 18:36:17.488386 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:20.545016) v2 -- ?+0 0x5beba40 con 0x61b0420 -6860> 2014-05-19 18:36:17.488417 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6859> 2014-05-19 18:36:17.488435 7fd04f907700 5 osd.0 21 tick -6858> 2014-05-19 18:36:17.488447 7fd04f907700 7 osd.0 21 do_mon_report -6857> 2014-05-19 18:36:17.488450 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -6856> 2014-05-19 18:36:17.488455 7fd04f907700 20 osd.0 21 send_pg_stats -6855> 2014-05-19 18:36:17.488489 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6854> 2014-05-19 18:36:17.488502 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.09 < max 0.5 = yes -6853> 2014-05-19 18:36:17.488519 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -6852> 2014-05-19 18:36:17.488524 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 220.213 < min (86400 seconds) -6851> 2014-05-19 18:36:17.488565 7fd04f907700 20 osd.0 21 sched_scrub done -6850> 2014-05-19 18:36:17.488571 7fd04f907700 10 osd.0 21 do_waiters -- start -6849> 2014-05-19 18:36:17.488595 7fd04f907700 10 osd.0 21 do_waiters -- finish -6848> 2014-05-19 18:36:17.810275 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:22.895894) v2 ==== 47+0+0 (704605378 0 0) 0x73c2000 con 0x30946e0 -6847> 2014-05-19 18:36:17.810276 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 60 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:22.895894) v2 ==== 47+0+0 (704605378 0 0) 0x5c04000 con 0x3094160 -6846> 2014-05-19 18:36:17.810324 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:22.895894) v2 -- ?+0 0x5c468c0 con 0x30946e0 -6845> 2014-05-19 18:36:17.810361 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6844> 2014-05-19 18:36:17.810390 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:22.895894) v2 -- ?+0 0x66f8540 con 0x3094160 -6843> 2014-05-19 18:36:17.810435 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6842> 2014-05-19 18:36:17.857346 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:20.914032) v2 ==== 47+0+0 (2607968376 0 0) 0x5beba40 con 0x4cb9340 -6841> 2014-05-19 18:36:17.857356 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:20.914032) v2 ==== 47+0+0 (2607968376 0 0) 0x5deb500 con 0x4cb8dc0 -6840> 2014-05-19 18:36:17.857409 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:20.914032) v2 -- ?+0 0x5c04000 con 0x4cb9340 -6839> 2014-05-19 18:36:17.857446 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6838> 2014-05-19 18:36:17.857466 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:20.914032) v2 -- ?+0 0x73c2000 con 0x4cb8dc0 -6837> 2014-05-19 18:36:17.857502 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6836> 2014-05-19 18:36:18.370121 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:23.455751) v2 ==== 47+0+0 (2064865395 0 0) 0x36f7180 con 0x31e1600 -6835> 2014-05-19 18:36:18.370150 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:23.455751) v2 ==== 47+0+0 (2064865395 0 0) 0x5beaa80 con 0x61b0160 -6834> 2014-05-19 18:36:18.370167 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:23.455751) v2 -- ?+0 0x5deb500 con 0x31e1600 -6833> 2014-05-19 18:36:18.370199 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6832> 2014-05-19 18:36:18.370225 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:23.455751) v2 -- ?+0 0x5beba40 con 0x61b0160 -6831> 2014-05-19 18:36:18.370278 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6830> 2014-05-19 18:36:18.488681 7fd04f907700 5 osd.0 21 tick -6829> 2014-05-19 18:36:18.488702 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -6828> 2014-05-19 18:36:18.488706 7fd04f907700 10 osd.0 21 do_waiters -- start -6827> 2014-05-19 18:36:18.488708 7fd04f907700 10 osd.0 21 do_waiters -- finish -6826> 2014-05-19 18:36:18.488750 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6825> 2014-05-19 18:36:18.773712 7fd04e905700 20 KeyValueStore::op_tp worker waiting -6824> 2014-05-19 18:36:18.773785 7fd043ebd700 20 OSD::op_tp worker waiting -6823> 2014-05-19 18:36:18.946048 7fd0396a8700 20 OSD::command_tp worker waiting -6822> 2014-05-19 18:36:19.083426 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:19.083224) v2 ==== 47+0+0 (3108730656 0 0) 0x79628c0 con 0x4cb86e0 -6821> 2014-05-19 18:36:19.083423 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:19.083224) v2 ==== 47+0+0 (3108730656 0 0) 0x5dff340 con 0x4cb8420 -6820> 2014-05-19 18:36:19.083474 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:19.083224) v2 -- ?+0 0x36f7180 con 0x4cb86e0 -6819> 2014-05-19 18:36:19.083518 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6818> 2014-05-19 18:36:19.083540 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:19.083224) v2 -- ?+0 0x5beaa80 con 0x4cb8420 -6817> 2014-05-19 18:36:19.083586 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6816> 2014-05-19 18:36:19.340267 7fd03c6ae700 20 OSD::op_tp worker waiting -6815> 2014-05-19 18:36:19.340291 7fd041eb9700 20 OSD::op_tp worker waiting -6814> 2014-05-19 18:36:19.340365 7fd03e6b2700 20 OSD::op_tp worker waiting -6813> 2014-05-19 18:36:19.340741 7fd03d6b0700 20 OSD::op_tp worker waiting -6812> 2014-05-19 18:36:19.340906 7fd0436bc700 20 OSD::op_tp worker waiting -6811> 2014-05-19 18:36:19.341033 7fd03bead700 20 OSD::op_tp worker waiting -6810> 2014-05-19 18:36:19.341110 7fd03f6b4700 20 OSD::op_tp worker waiting -6809> 2014-05-19 18:36:19.354954 7fd0446be700 20 OSD::op_tp worker waiting -6808> 2014-05-19 18:36:19.354993 7fd03feb5700 20 OSD::op_tp worker waiting -6807> 2014-05-19 18:36:19.361983 7fd040eb7700 20 OSD::op_tp worker waiting -6806> 2014-05-19 18:36:19.367236 7fd0406b6700 20 OSD::op_tp worker waiting -6805> 2014-05-19 18:36:19.388767 7fd039ea9700 20 OSD::disk_tp worker waiting -6804> 2014-05-19 18:36:19.392369 7fd03aeab700 20 OSD::op_tp worker waiting -6803> 2014-05-19 18:36:19.392390 7fd03eeb3700 20 OSD::op_tp worker waiting -6802> 2014-05-19 18:36:19.410348 7fd042ebb700 20 OSD::op_tp worker waiting -6801> 2014-05-19 18:36:19.410371 7fd03deb1700 20 OSD::op_tp worker waiting -6800> 2014-05-19 18:36:19.418896 7fd03b6ac700 20 OSD::op_tp worker waiting -6799> 2014-05-19 18:36:19.418895 7fd04e104700 20 KeyValueStore::op_tp worker waiting -6798> 2014-05-19 18:36:19.418924 7fd0426ba700 20 OSD::op_tp worker waiting -6797> 2014-05-19 18:36:19.418947 7fd0416b8700 20 OSD::op_tp worker waiting -6796> 2014-05-19 18:36:19.418963 7fd03ceaf700 20 OSD::op_tp worker waiting -6795> 2014-05-19 18:36:19.488794 7fd04f907700 5 osd.0 21 tick -6794> 2014-05-19 18:36:19.488842 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.08 < max 0.5 = yes -6793> 2014-05-19 18:36:19.488855 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -6792> 2014-05-19 18:36:19.488863 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6791> 2014-05-19 18:36:19.488858 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 222.213 < min (86400 seconds) -6790> 2014-05-19 18:36:19.488876 7fd04f907700 20 osd.0 21 sched_scrub done -6789> 2014-05-19 18:36:19.488881 7fd04f907700 10 osd.0 21 do_waiters -- start -6788> 2014-05-19 18:36:19.488883 7fd04f907700 10 osd.0 21 do_waiters -- finish -6787> 2014-05-19 18:36:19.788748 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:22.845523) v2 ==== 47+0+0 (4136039867 0 0) 0x5c46700 con 0x61ae160 -6786> 2014-05-19 18:36:19.788794 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:22.845523) v2 -- ?+0 0x79628c0 con 0x61ae160 -6785> 2014-05-19 18:36:19.788809 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:22.845523) v2 ==== 47+0+0 (4136039867 0 0) 0x66f9a40 con 0x61b0420 -6784> 2014-05-19 18:36:19.788874 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6783> 2014-05-19 18:36:19.788898 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:22.845523) v2 -- ?+0 0x5dff340 con 0x61b0420 -6782> 2014-05-19 18:36:19.788941 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6781> 2014-05-19 18:36:19.907010 7fd0456c0700 10 monclient: tick -6780> 2014-05-19 18:36:19.907023 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:35:49.907021) -6779> 2014-05-19 18:36:19.907041 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:36:19.907041; renew after: 2014-05-19 18:38:19.906664) -- no -6778> 2014-05-19 18:36:20.157927 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:23.214673) v2 ==== 47+0+0 (4170240883 0 0) 0x7a36fc0 con 0x4cb9340 -6777> 2014-05-19 18:36:20.157960 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:23.214673) v2 ==== 47+0+0 (4170240883 0 0) 0x5deaa80 con 0x4cb8dc0 -6776> 2014-05-19 18:36:20.157980 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:23.214673) v2 -- ?+0 0x66f9a40 con 0x4cb9340 -6775> 2014-05-19 18:36:20.158037 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6774> 2014-05-19 18:36:20.158108 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:23.214673) v2 -- ?+0 0x5c46700 con 0x4cb8dc0 -6773> 2014-05-19 18:36:20.158139 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6772> 2014-05-19 18:36:20.488964 7fd04f907700 5 osd.0 21 tick -6771> 2014-05-19 18:36:20.488987 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -6770> 2014-05-19 18:36:20.488993 7fd04f907700 10 osd.0 21 do_waiters -- start -6769> 2014-05-19 18:36:20.488998 7fd04f907700 10 osd.0 21 do_waiters -- finish -6768> 2014-05-19 18:36:20.489024 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6767> 2014-05-19 18:36:20.670510 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:25.756221) v2 ==== 47+0+0 (2402575938 0 0) 0x36f7880 con 0x31e1600 -6766> 2014-05-19 18:36:20.670509 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:25.756221) v2 ==== 47+0+0 (2402575938 0 0) 0x5beb340 con 0x61b0160 -6765> 2014-05-19 18:36:20.670565 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.756221) v2 -- ?+0 0x5deaa80 con 0x31e1600 -6764> 2014-05-19 18:36:20.670642 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6763> 2014-05-19 18:36:20.670669 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.756221) v2 -- ?+0 0x7a36fc0 con 0x61b0160 -6762> 2014-05-19 18:36:20.670707 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6761> 2014-05-19 18:36:20.773800 7fd04e905700 20 KeyValueStore::op_tp worker waiting -6760> 2014-05-19 18:36:20.773863 7fd043ebd700 20 OSD::op_tp worker waiting -6759> 2014-05-19 18:36:20.946143 7fd0396a8700 20 OSD::command_tp worker waiting -6758> 2014-05-19 18:36:21.153587 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -6757> 2014-05-19 18:36:21.153606 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -6756> 2014-05-19 18:36:21.153614 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x692d180 con 0x3622580 -6755> 2014-05-19 18:36:21.153639 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x5f0ba40 con 0x3622840 -6754> 2014-05-19 18:36:21.153677 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x63781c0 con 0x3622b00 -6753> 2014-05-19 18:36:21.153692 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x7733c00 con 0x3622dc0 -6752> 2014-05-19 18:36:21.153705 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x7999180 con 0x30594a0 -6751> 2014-05-19 18:36:21.153731 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x682fc00 con 0x3622f20 -6750> 2014-05-19 18:36:21.153761 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x6e2a380 con 0x5e7ef20 -6749> 2014-05-19 18:36:21.153786 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x76ac000 con 0x5e7f1e0 -6748> 2014-05-19 18:36:21.153812 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x64ae8c0 con 0x5e7e2c0 -6747> 2014-05-19 18:36:21.153836 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x51a7dc0 con 0x5d7bb80 -6746> 2014-05-19 18:36:21.153859 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x36f6000 con 0x5e7ec60 -6745> 2014-05-19 18:36:21.153884 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x60a0e00 con 0x531b760 -6744> 2014-05-19 18:36:21.153910 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x73c0540 con 0x53bba20 -6743> 2014-05-19 18:36:21.153936 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x6c4b180 con 0x51cac60 -6742> 2014-05-19 18:36:21.153962 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x692d500 con 0x5bd66e0 -6741> 2014-05-19 18:36:21.153981 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:21.153612) v2 -- ?+0 0x5519dc0 con 0x5bd6840 -6740> 2014-05-19 18:36:21.153978 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x669e540 con 0x3622580 -6739> 2014-05-19 18:36:21.154030 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6738> 2014-05-19 18:36:21.154086 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x5c05340 con 0x3622dc0 -6737> 2014-05-19 18:36:21.154123 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6736> 2014-05-19 18:36:21.154142 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x68ab500 con 0x3622b00 -6735> 2014-05-19 18:36:21.154159 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6734> 2014-05-19 18:36:21.154165 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x72e41c0 con 0x3622840 -6733> 2014-05-19 18:36:21.154186 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6732> 2014-05-19 18:36:21.154194 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x511ddc0 con 0x3622f20 -6731> 2014-05-19 18:36:21.154235 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6730> 2014-05-19 18:36:21.154250 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x5f7b340 con 0x5e7e2c0 -6729> 2014-05-19 18:36:21.154281 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6728> 2014-05-19 18:36:21.154327 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x77461c0 con 0x5e7f1e0 -6727> 2014-05-19 18:36:21.154357 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6726> 2014-05-19 18:36:21.154380 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x6b71c00 con 0x5d7bb80 -6725> 2014-05-19 18:36:21.154397 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6724> 2014-05-19 18:36:21.154405 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x7971dc0 con 0x5e7ec60 -6723> 2014-05-19 18:36:21.154419 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6722> 2014-05-19 18:36:21.154427 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x5667c00 con 0x53bba20 -6721> 2014-05-19 18:36:21.154442 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6720> 2014-05-19 18:36:21.154450 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x74a76c0 con 0x51cac60 -6719> 2014-05-19 18:36:21.154468 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6718> 2014-05-19 18:36:21.154476 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x7122540 con 0x5bd66e0 -6717> 2014-05-19 18:36:21.154490 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6716> 2014-05-19 18:36:21.154498 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 65 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x36f68c0 con 0x5bd6840 -6715> 2014-05-19 18:36:21.154514 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6714> 2014-05-19 18:36:21.154520 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x51a6540 con 0x30594a0 -6713> 2014-05-19 18:36:21.154537 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6712> 2014-05-19 18:36:21.154542 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x6008380 con 0x5e7ef20 -6711> 2014-05-19 18:36:21.154555 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6710> 2014-05-19 18:36:21.154561 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:21.153612) v2 ==== 47+0+0 (3655050311 0 0) 0x6c4c540 con 0x531b760 -6709> 2014-05-19 18:36:21.154578 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6708> 2014-05-19 18:36:21.340339 7fd03c6ae700 20 OSD::op_tp worker waiting -6707> 2014-05-19 18:36:21.340366 7fd041eb9700 20 OSD::op_tp worker waiting -6706> 2014-05-19 18:36:21.340457 7fd03e6b2700 20 OSD::op_tp worker waiting -6705> 2014-05-19 18:36:21.340834 7fd03d6b0700 20 OSD::op_tp worker waiting -6704> 2014-05-19 18:36:21.340975 7fd0436bc700 20 OSD::op_tp worker waiting -6703> 2014-05-19 18:36:21.341099 7fd03bead700 20 OSD::op_tp worker waiting -6702> 2014-05-19 18:36:21.341182 7fd03f6b4700 20 OSD::op_tp worker waiting -6701> 2014-05-19 18:36:21.355036 7fd0446be700 20 OSD::op_tp worker waiting -6700> 2014-05-19 18:36:21.355072 7fd03feb5700 20 OSD::op_tp worker waiting -6699> 2014-05-19 18:36:21.362080 7fd040eb7700 20 OSD::op_tp worker waiting -6698> 2014-05-19 18:36:21.367332 7fd0406b6700 20 OSD::op_tp worker waiting -6697> 2014-05-19 18:36:21.388847 7fd039ea9700 20 OSD::disk_tp worker waiting -6696> 2014-05-19 18:36:21.392468 7fd03aeab700 20 OSD::op_tp worker waiting -6695> 2014-05-19 18:36:21.392495 7fd03eeb3700 20 OSD::op_tp worker waiting -6694> 2014-05-19 18:36:21.410448 7fd042ebb700 20 OSD::op_tp worker waiting -6693> 2014-05-19 18:36:21.410472 7fd03deb1700 20 OSD::op_tp worker waiting -6692> 2014-05-19 18:36:21.418996 7fd04e104700 20 KeyValueStore::op_tp worker waiting -6691> 2014-05-19 18:36:21.418998 7fd03b6ac700 20 OSD::op_tp worker waiting -6690> 2014-05-19 18:36:21.419065 7fd0426ba700 20 OSD::op_tp worker waiting -6689> 2014-05-19 18:36:21.419124 7fd03ceaf700 20 OSD::op_tp worker waiting -6688> 2014-05-19 18:36:21.419149 7fd0416b8700 20 OSD::op_tp worker waiting -6687> 2014-05-19 18:36:21.489085 7fd04f907700 5 osd.0 21 tick -6686> 2014-05-19 18:36:21.489107 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -6685> 2014-05-19 18:36:21.489111 7fd04f907700 10 osd.0 21 do_waiters -- start -6684> 2014-05-19 18:36:21.489113 7fd04f907700 10 osd.0 21 do_waiters -- finish -6683> 2014-05-19 18:36:21.489138 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6682> 2014-05-19 18:36:22.089261 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:25.145996) v2 ==== 47+0+0 (3518967851 0 0) 0x612ce00 con 0x61ae160 -6681> 2014-05-19 18:36:22.089279 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:25.145996) v2 ==== 47+0+0 (3518967851 0 0) 0x6543500 con 0x61b0420 -6680> 2014-05-19 18:36:22.089302 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.145996) v2 -- ?+0 0x36f7880 con 0x61ae160 -6679> 2014-05-19 18:36:22.089337 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6678> 2014-05-19 18:36:22.089362 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.145996) v2 -- ?+0 0x5beb340 con 0x61b0420 -6677> 2014-05-19 18:36:22.089393 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6676> 2014-05-19 18:36:22.185686 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:27.271472) v2 ==== 47+0+0 (254880749 0 0) 0x2ee08c0 con 0x5b21600 -6675> 2014-05-19 18:36:22.185725 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:27.271472) v2 ==== 47+0+0 (254880749 0 0) 0x5cbc540 con 0x5f5b760 -6674> 2014-05-19 18:36:22.185737 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:27.271472) v2 -- ?+0 0x612ce00 con 0x5b21600 -6673> 2014-05-19 18:36:22.185766 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6672> 2014-05-19 18:36:22.185788 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:27.271472) v2 -- ?+0 0x6543500 con 0x5f5b760 -6671> 2014-05-19 18:36:22.185822 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6670> 2014-05-19 18:36:22.489199 7fd04f907700 5 osd.0 21 tick -6669> 2014-05-19 18:36:22.489216 7fd04f907700 7 osd.0 21 do_mon_report -6668> 2014-05-19 18:36:22.489220 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -6667> 2014-05-19 18:36:22.489223 7fd04f907700 20 osd.0 21 send_pg_stats -6666> 2014-05-19 18:36:22.489231 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -6665> 2014-05-19 18:36:22.489234 7fd04f907700 10 osd.0 21 do_waiters -- start -6664> 2014-05-19 18:36:22.489236 7fd04f907700 10 osd.0 21 do_waiters -- finish -6663> 2014-05-19 18:36:22.489276 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6662> 2014-05-19 18:36:22.773876 7fd04e905700 20 KeyValueStore::op_tp worker waiting -6661> 2014-05-19 18:36:22.773940 7fd043ebd700 20 OSD::op_tp worker waiting -6660> 2014-05-19 18:36:22.919692 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:25.976441) v2 ==== 47+0+0 (2857584774 0 0) 0x77e8c40 con 0x61ae420 -6659> 2014-05-19 18:36:22.919745 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.976441) v2 -- ?+0 0x2ee08c0 con 0x61ae420 -6658> 2014-05-19 18:36:22.919779 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6657> 2014-05-19 18:36:22.919692 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:25.976441) v2 ==== 47+0+0 (2857584774 0 0) 0x7746fc0 con 0x618fe40 -6656> 2014-05-19 18:36:22.919794 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.976441) v2 -- ?+0 0x5cbc540 con 0x618fe40 -6655> 2014-05-19 18:36:22.919824 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6654> 2014-05-19 18:36:22.946237 7fd0396a8700 20 OSD::command_tp worker waiting -6653> 2014-05-19 18:36:22.970953 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:28.056747) v2 ==== 47+0+0 (1188776182 0 0) 0x5bea1c0 con 0x61b0160 -6652> 2014-05-19 18:36:22.970965 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:28.056747) v2 ==== 47+0+0 (1188776182 0 0) 0x71221c0 con 0x31e1600 -6651> 2014-05-19 18:36:22.971011 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.056747) v2 -- ?+0 0x7746fc0 con 0x61b0160 -6650> 2014-05-19 18:36:22.971085 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6649> 2014-05-19 18:36:22.971123 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.056747) v2 -- ?+0 0x77e8c40 con 0x31e1600 -6648> 2014-05-19 18:36:22.971160 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6647> 2014-05-19 18:36:23.069193 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:23.069026) v2 ==== 47+0+0 (3330662916 0 0) 0x77e8000 con 0x4cb8160 -6646> 2014-05-19 18:36:23.069242 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:23.069026) v2 -- ?+0 0x71221c0 con 0x4cb8160 -6645> 2014-05-19 18:36:23.069242 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:23.069026) v2 ==== 47+0+0 (3330662916 0 0) 0x5519c00 con 0x4cb82c0 -6644> 2014-05-19 18:36:23.069278 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6643> 2014-05-19 18:36:23.069348 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:23.069026) v2 -- ?+0 0x5bea1c0 con 0x4cb82c0 -6642> 2014-05-19 18:36:23.069457 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6641> 2014-05-19 18:36:23.110471 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:28.196359) v2 ==== 47+0+0 (324957002 0 0) 0x55c9a40 con 0x3094160 -6640> 2014-05-19 18:36:23.110481 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 61 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:28.196359) v2 ==== 47+0+0 (324957002 0 0) 0x70cc1c0 con 0x30946e0 -6639> 2014-05-19 18:36:23.110518 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.196359) v2 -- ?+0 0x5519c00 con 0x3094160 -6638> 2014-05-19 18:36:23.110552 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6637> 2014-05-19 18:36:23.110575 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.196359) v2 -- ?+0 0x77e8000 con 0x30946e0 -6636> 2014-05-19 18:36:23.110651 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6635> 2014-05-19 18:36:23.340421 7fd03c6ae700 20 OSD::op_tp worker waiting -6634> 2014-05-19 18:36:23.340449 7fd041eb9700 20 OSD::op_tp worker waiting -6633> 2014-05-19 18:36:23.340527 7fd03e6b2700 20 OSD::op_tp worker waiting -6632> 2014-05-19 18:36:23.340915 7fd03d6b0700 20 OSD::op_tp worker waiting -6631> 2014-05-19 18:36:23.341053 7fd0436bc700 20 OSD::op_tp worker waiting -6630> 2014-05-19 18:36:23.341171 7fd03bead700 20 OSD::op_tp worker waiting -6629> 2014-05-19 18:36:23.341260 7fd03f6b4700 20 OSD::op_tp worker waiting -6628> 2014-05-19 18:36:23.355119 7fd0446be700 20 OSD::op_tp worker waiting -6627> 2014-05-19 18:36:23.355149 7fd03feb5700 20 OSD::op_tp worker waiting -6626> 2014-05-19 18:36:23.362165 7fd040eb7700 20 OSD::op_tp worker waiting -6625> 2014-05-19 18:36:23.367428 7fd0406b6700 20 OSD::op_tp worker waiting -6624> 2014-05-19 18:36:23.388936 7fd039ea9700 20 OSD::disk_tp worker waiting -6623> 2014-05-19 18:36:23.392567 7fd03aeab700 20 OSD::op_tp worker waiting -6622> 2014-05-19 18:36:23.392621 7fd03eeb3700 20 OSD::op_tp worker waiting -6621> 2014-05-19 18:36:23.410530 7fd042ebb700 20 OSD::op_tp worker waiting -6620> 2014-05-19 18:36:23.410550 7fd03deb1700 20 OSD::op_tp worker waiting -6619> 2014-05-19 18:36:23.419082 7fd04e104700 20 KeyValueStore::op_tp worker waiting -6618> 2014-05-19 18:36:23.419098 7fd03b6ac700 20 OSD::op_tp worker waiting -6617> 2014-05-19 18:36:23.419184 7fd0426ba700 20 OSD::op_tp worker waiting -6616> 2014-05-19 18:36:23.419224 7fd0416b8700 20 OSD::op_tp worker waiting -6615> 2014-05-19 18:36:23.419246 7fd03ceaf700 20 OSD::op_tp worker waiting -6614> 2014-05-19 18:36:23.489324 7fd04f907700 5 osd.0 21 tick -6613> 2014-05-19 18:36:23.489345 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -6612> 2014-05-19 18:36:23.489350 7fd04f907700 10 osd.0 21 do_waiters -- start -6611> 2014-05-19 18:36:23.489352 7fd04f907700 10 osd.0 21 do_waiters -- finish -6610> 2014-05-19 18:36:23.489375 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6609> 2014-05-19 18:36:24.486032 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:29.571920) v2 ==== 47+0+0 (2569818060 0 0) 0x662b180 con 0x5b21600 -6608> 2014-05-19 18:36:24.486071 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:29.571920) v2 ==== 47+0+0 (2569818060 0 0) 0x68aae00 con 0x5f5b760 -6607> 2014-05-19 18:36:24.486113 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:29.571920) v2 -- ?+0 0x70cc1c0 con 0x5b21600 -6606> 2014-05-19 18:36:24.486192 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6605> 2014-05-19 18:36:24.486263 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:29.571920) v2 -- ?+0 0x55c9a40 con 0x5f5b760 -6604> 2014-05-19 18:36:24.486306 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6603> 2014-05-19 18:36:24.489439 7fd04f907700 5 osd.0 21 tick -6602> 2014-05-19 18:36:24.489487 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes -6601> 2014-05-19 18:36:24.489500 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -6600> 2014-05-19 18:36:24.489508 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6599> 2014-05-19 18:36:24.489503 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 227.214 < min (86400 seconds) -6598> 2014-05-19 18:36:24.489522 7fd04f907700 20 osd.0 21 sched_scrub done -6597> 2014-05-19 18:36:24.489525 7fd04f907700 10 osd.0 21 do_waiters -- start -6596> 2014-05-19 18:36:24.489527 7fd04f907700 10 osd.0 21 do_waiters -- finish -6595> 2014-05-19 18:36:24.654107 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -6594> 2014-05-19 18:36:24.654125 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -6593> 2014-05-19 18:36:24.654133 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x7999dc0 con 0x3622580 -6592> 2014-05-19 18:36:24.654159 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x7733dc0 con 0x3622840 -6591> 2014-05-19 18:36:24.654197 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x60f61c0 con 0x3622b00 -6590> 2014-05-19 18:36:24.654212 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x6009340 con 0x3622dc0 -6589> 2014-05-19 18:36:24.654226 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x83881c0 con 0x30594a0 -6588> 2014-05-19 18:36:24.654253 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x6b71500 con 0x3622f20 -6587> 2014-05-19 18:36:24.654283 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x711e700 con 0x5e7ef20 -6586> 2014-05-19 18:36:24.654309 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x6fe7a40 con 0x5e7f1e0 -6585> 2014-05-19 18:36:24.654334 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x64aec40 con 0x5e7e2c0 -6584> 2014-05-19 18:36:24.654360 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x2f4fdc0 con 0x5d7bb80 -6583> 2014-05-19 18:36:24.654383 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x682e540 con 0x5e7ec60 -6582> 2014-05-19 18:36:24.654407 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x36f6700 con 0x531b760 -6581> 2014-05-19 18:36:24.654431 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x60a0fc0 con 0x53bba20 -6580> 2014-05-19 18:36:24.654455 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x7a5f340 con 0x51cac60 -6579> 2014-05-19 18:36:24.654479 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x55f4380 con 0x5bd66e0 -6578> 2014-05-19 18:36:24.654504 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:24.654132) v2 -- ?+0 0x692dc00 con 0x5bd6840 -6577> 2014-05-19 18:36:24.654547 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x60e2000 con 0x3622580 -6576> 2014-05-19 18:36:24.654597 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6575> 2014-05-19 18:36:24.654612 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x5c05500 con 0x3622dc0 -6574> 2014-05-19 18:36:24.654662 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6573> 2014-05-19 18:36:24.654691 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x60e0700 con 0x3622840 -6572> 2014-05-19 18:36:24.654713 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6571> 2014-05-19 18:36:24.654725 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x59bcfc0 con 0x3622b00 -6570> 2014-05-19 18:36:24.654740 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6569> 2014-05-19 18:36:24.654746 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x725d880 con 0x5e7ef20 -6568> 2014-05-19 18:36:24.654863 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6567> 2014-05-19 18:36:24.654895 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x5f7ac40 con 0x5e7e2c0 -6566> 2014-05-19 18:36:24.654923 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6565> 2014-05-19 18:36:24.654939 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x8388700 con 0x5d7bb80 -6564> 2014-05-19 18:36:24.654957 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6563> 2014-05-19 18:36:24.654965 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x7970540 con 0x5e7ec60 -6562> 2014-05-19 18:36:24.654980 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6561> 2014-05-19 18:36:24.654989 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x6c4d6c0 con 0x531b760 -6560> 2014-05-19 18:36:24.655003 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6559> 2014-05-19 18:36:24.655011 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x7980380 con 0x53bba20 -6558> 2014-05-19 18:36:24.655028 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6557> 2014-05-19 18:36:24.655036 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x5dfee00 con 0x51cac60 -6556> 2014-05-19 18:36:24.655053 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6555> 2014-05-19 18:36:24.655059 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x7122c40 con 0x5bd66e0 -6554> 2014-05-19 18:36:24.655074 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6553> 2014-05-19 18:36:24.655082 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 66 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x5cbc380 con 0x5bd6840 -6552> 2014-05-19 18:36:24.655097 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6551> 2014-05-19 18:36:24.655105 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x6504700 con 0x30594a0 -6550> 2014-05-19 18:36:24.655119 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6549> 2014-05-19 18:36:24.655126 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x6113880 con 0x3622f20 -6548> 2014-05-19 18:36:24.655140 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6547> 2014-05-19 18:36:24.655148 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.654132) v2 ==== 47+0+0 (2625945897 0 0) 0x7747880 con 0x5e7f1e0 -6546> 2014-05-19 18:36:24.655165 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6545> 2014-05-19 18:36:24.773969 7fd04e905700 20 KeyValueStore::op_tp worker waiting -6544> 2014-05-19 18:36:24.774016 7fd043ebd700 20 OSD::op_tp worker waiting -6543> 2014-05-19 18:36:24.946330 7fd0396a8700 20 OSD::command_tp worker waiting -6542> 2014-05-19 18:36:24.983924 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:24.983750) v2 ==== 47+0+0 (4212594386 0 0) 0x75b08c0 con 0x4cb8420 -6541> 2014-05-19 18:36:24.983957 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:24.983750) v2 ==== 47+0+0 (4212594386 0 0) 0x5497dc0 con 0x4cb86e0 -6540> 2014-05-19 18:36:24.983973 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.983750) v2 -- ?+0 0x68aae00 con 0x4cb8420 -6539> 2014-05-19 18:36:24.984033 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6538> 2014-05-19 18:36:24.984102 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:24.983750) v2 -- ?+0 0x662b180 con 0x4cb86e0 -6537> 2014-05-19 18:36:24.984147 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6536> 2014-05-19 18:36:25.154647 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -6535> 2014-05-19 18:36:25.154666 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -6534> 2014-05-19 18:36:25.154674 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x5f0b180 con 0x3622580 -6533> 2014-05-19 18:36:25.154700 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x6710fc0 con 0x3622840 -6532> 2014-05-19 18:36:25.154737 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x7998700 con 0x3622b00 -6531> 2014-05-19 18:36:25.154752 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x7733a40 con 0x3622dc0 -6530> 2014-05-19 18:36:25.154784 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x6d61180 con 0x30594a0 -6529> 2014-05-19 18:36:25.154816 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x7178c40 con 0x3622f20 -6528> 2014-05-19 18:36:25.154843 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x2f4efc0 con 0x5e7ef20 -6527> 2014-05-19 18:36:25.154867 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x64afdc0 con 0x5e7f1e0 -6526> 2014-05-19 18:36:25.154891 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x76ac1c0 con 0x5e7e2c0 -6525> 2014-05-19 18:36:25.154917 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x72dfdc0 con 0x5d7bb80 -6524> 2014-05-19 18:36:25.154940 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x77321c0 con 0x5e7ec60 -6523> 2014-05-19 18:36:25.154963 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x36f7500 con 0x531b760 -6522> 2014-05-19 18:36:25.154987 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x60a1180 con 0x53bba20 -6521> 2014-05-19 18:36:25.155014 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x7a5f180 con 0x51cac60 -6520> 2014-05-19 18:36:25.155038 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x55f48c0 con 0x5bd66e0 -6519> 2014-05-19 18:36:25.155064 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:25.154672) v2 -- ?+0 0x692c000 con 0x5bd6840 -6518> 2014-05-19 18:36:25.155101 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x59bd500 con 0x3622b00 -6517> 2014-05-19 18:36:25.155174 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6516> 2014-05-19 18:36:25.155192 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x66f9dc0 con 0x3622580 -6515> 2014-05-19 18:36:25.155244 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6514> 2014-05-19 18:36:25.155279 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x60e01c0 con 0x3622840 -6513> 2014-05-19 18:36:25.155340 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6512> 2014-05-19 18:36:25.155363 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x5c05a40 con 0x3622dc0 -6511> 2014-05-19 18:36:25.155384 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6510> 2014-05-19 18:36:25.155394 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x7746c40 con 0x5e7f1e0 -6509> 2014-05-19 18:36:25.155409 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6508> 2014-05-19 18:36:25.155418 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x5f7a8c0 con 0x5e7e2c0 -6507> 2014-05-19 18:36:25.155433 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6506> 2014-05-19 18:36:25.155438 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x8388540 con 0x5d7bb80 -6505> 2014-05-19 18:36:25.155452 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6504> 2014-05-19 18:36:25.155481 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x7971500 con 0x5e7ec60 -6503> 2014-05-19 18:36:25.155506 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6502> 2014-05-19 18:36:25.155539 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x772d180 con 0x53bba20 -6501> 2014-05-19 18:36:25.155563 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6500> 2014-05-19 18:36:25.155572 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x2ee0fc0 con 0x51cac60 -6499> 2014-05-19 18:36:25.155586 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6498> 2014-05-19 18:36:25.155594 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x649e540 con 0x5bd66e0 -6497> 2014-05-19 18:36:25.155611 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6496> 2014-05-19 18:36:25.155617 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 67 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x70ea540 con 0x5bd6840 -6495> 2014-05-19 18:36:25.155633 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6494> 2014-05-19 18:36:25.155642 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x6504e00 con 0x30594a0 -6493> 2014-05-19 18:36:25.155661 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6492> 2014-05-19 18:36:25.155666 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x6113180 con 0x3622f20 -6491> 2014-05-19 18:36:25.155680 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6490> 2014-05-19 18:36:25.155685 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x725ca80 con 0x5e7ef20 -6489> 2014-05-19 18:36:25.155710 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6488> 2014-05-19 18:36:25.155718 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:25.154672) v2 ==== 47+0+0 (4177122688 0 0) 0x6c4cc40 con 0x531b760 -6487> 2014-05-19 18:36:25.155736 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6486> 2014-05-19 18:36:25.340495 7fd03c6ae700 20 OSD::op_tp worker waiting -6485> 2014-05-19 18:36:25.340523 7fd041eb9700 20 OSD::op_tp worker waiting -6484> 2014-05-19 18:36:25.340592 7fd03e6b2700 20 OSD::op_tp worker waiting -6483> 2014-05-19 18:36:25.340996 7fd03d6b0700 20 OSD::op_tp worker waiting -6482> 2014-05-19 18:36:25.341118 7fd0436bc700 20 OSD::op_tp worker waiting -6481> 2014-05-19 18:36:25.341238 7fd03bead700 20 OSD::op_tp worker waiting -6480> 2014-05-19 18:36:25.341341 7fd03f6b4700 20 OSD::op_tp worker waiting -6479> 2014-05-19 18:36:25.355207 7fd0446be700 20 OSD::op_tp worker waiting -6478> 2014-05-19 18:36:25.355227 7fd03feb5700 20 OSD::op_tp worker waiting -6477> 2014-05-19 18:36:25.362248 7fd040eb7700 20 OSD::op_tp worker waiting -6476> 2014-05-19 18:36:25.367526 7fd0406b6700 20 OSD::op_tp worker waiting -6475> 2014-05-19 18:36:25.389031 7fd039ea9700 20 OSD::disk_tp worker waiting -6474> 2014-05-19 18:36:25.392679 7fd03aeab700 20 OSD::op_tp worker waiting -6473> 2014-05-19 18:36:25.392700 7fd03eeb3700 20 OSD::op_tp worker waiting -6472> 2014-05-19 18:36:25.410628 7fd042ebb700 20 OSD::op_tp worker waiting -6471> 2014-05-19 18:36:25.410654 7fd03deb1700 20 OSD::op_tp worker waiting -6470> 2014-05-19 18:36:25.410845 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:30.496804) v2 ==== 47+0+0 (1978536715 0 0) 0x6009880 con 0x30946e0 -6469> 2014-05-19 18:36:25.410895 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:30.496804) v2 -- ?+0 0x5497dc0 con 0x30946e0 -6468> 2014-05-19 18:36:25.410936 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6467> 2014-05-19 18:36:25.410900 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 62 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:30.496804) v2 ==== 47+0+0 (1978536715 0 0) 0x55c9500 con 0x3094160 -6466> 2014-05-19 18:36:25.410984 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:30.496804) v2 -- ?+0 0x75b08c0 con 0x3094160 -6465> 2014-05-19 18:36:25.411015 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6464> 2014-05-19 18:36:25.419169 7fd04e104700 20 KeyValueStore::op_tp worker waiting -6463> 2014-05-19 18:36:25.419176 7fd03b6ac700 20 OSD::op_tp worker waiting -6462> 2014-05-19 18:36:25.419272 7fd0426ba700 20 OSD::op_tp worker waiting -6461> 2014-05-19 18:36:25.419301 7fd0416b8700 20 OSD::op_tp worker waiting -6460> 2014-05-19 18:36:25.419313 7fd03ceaf700 20 OSD::op_tp worker waiting -6459> 2014-05-19 18:36:25.489613 7fd04f907700 5 osd.0 21 tick -6458> 2014-05-19 18:36:25.489665 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6457> 2014-05-19 18:36:25.489661 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes -6456> 2014-05-19 18:36:25.489675 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -6455> 2014-05-19 18:36:25.489678 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 228.214 < min (86400 seconds) -6454> 2014-05-19 18:36:25.489697 7fd04f907700 20 osd.0 21 sched_scrub done -6453> 2014-05-19 18:36:25.489700 7fd04f907700 10 osd.0 21 do_waiters -- start -6452> 2014-05-19 18:36:25.489702 7fd04f907700 10 osd.0 21 do_waiters -- finish -6451> 2014-05-19 18:36:26.058321 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:29.115153) v2 ==== 47+0+0 (3860503651 0 0) 0x7947dc0 con 0x4cb8dc0 -6450> 2014-05-19 18:36:26.058340 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:29.115153) v2 ==== 47+0+0 (3860503651 0 0) 0x79e6540 con 0x4cb9340 -6449> 2014-05-19 18:36:26.058371 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:29.115153) v2 -- ?+0 0x6009880 con 0x4cb8dc0 -6448> 2014-05-19 18:36:26.058404 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6447> 2014-05-19 18:36:26.058422 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:29.115153) v2 -- ?+0 0x55c9500 con 0x4cb9340 -6446> 2014-05-19 18:36:26.058456 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6445> 2014-05-19 18:36:26.420177 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:29.476994) v2 ==== 47+0+0 (3930926623 0 0) 0x77e8e00 con 0x61ae420 -6444> 2014-05-19 18:36:26.420253 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:29.476994) v2 -- ?+0 0x7947dc0 con 0x61ae420 -6443> 2014-05-19 18:36:26.420282 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6442> 2014-05-19 18:36:26.420275 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:29.476994) v2 ==== 47+0+0 (3930926623 0 0) 0x669f340 con 0x618fe40 -6441> 2014-05-19 18:36:26.420321 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:29.476994) v2 -- ?+0 0x79e6540 con 0x618fe40 -6440> 2014-05-19 18:36:26.420369 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6439> 2014-05-19 18:36:26.489784 7fd04f907700 5 osd.0 21 tick -6438> 2014-05-19 18:36:26.489832 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes -6437> 2014-05-19 18:36:26.489845 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -6436> 2014-05-19 18:36:26.489853 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6435> 2014-05-19 18:36:26.489848 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 229.214 < min (86400 seconds) -6434> 2014-05-19 18:36:26.489866 7fd04f907700 20 osd.0 21 sched_scrub done -6433> 2014-05-19 18:36:26.489869 7fd04f907700 10 osd.0 21 do_waiters -- start -6432> 2014-05-19 18:36:26.489871 7fd04f907700 10 osd.0 21 do_waiters -- finish -6431> 2014-05-19 18:36:26.569762 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:26.569589) v2 ==== 47+0+0 (3309077702 0 0) 0x5519340 con 0x4cb82c0 -6430> 2014-05-19 18:36:26.569777 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:26.569589) v2 ==== 47+0+0 (3309077702 0 0) 0x77e96c0 con 0x4cb8160 -6429> 2014-05-19 18:36:26.569804 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:26.569589) v2 -- ?+0 0x669f340 con 0x4cb82c0 -6428> 2014-05-19 18:36:26.569828 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6427> 2014-05-19 18:36:26.569852 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:26.569589) v2 -- ?+0 0x77e8e00 con 0x4cb8160 -6426> 2014-05-19 18:36:26.569943 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6425> 2014-05-19 18:36:26.774062 7fd04e905700 20 KeyValueStore::op_tp worker waiting -6424> 2014-05-19 18:36:26.774095 7fd043ebd700 20 OSD::op_tp worker waiting -6423> 2014-05-19 18:36:26.920876 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:29.977682) v2 ==== 47+0+0 (247443720 0 0) 0x5c05880 con 0x618fe40 -6422> 2014-05-19 18:36:26.920884 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:29.977682) v2 ==== 47+0+0 (247443720 0 0) 0x788b500 con 0x61ae420 -6421> 2014-05-19 18:36:26.920922 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:29.977682) v2 -- ?+0 0x5519340 con 0x618fe40 -6420> 2014-05-19 18:36:26.920957 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6419> 2014-05-19 18:36:26.921003 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:29.977682) v2 -- ?+0 0x77e96c0 con 0x61ae420 -6418> 2014-05-19 18:36:26.921045 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6417> 2014-05-19 18:36:26.946425 7fd0396a8700 20 OSD::command_tp worker waiting -6416> 2014-05-19 18:36:27.070326 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:27.070162) v2 ==== 47+0+0 (1861970833 0 0) 0x77e81c0 con 0x4cb8160 -6415> 2014-05-19 18:36:27.070408 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:27.070162) v2 -- ?+0 0x788b500 con 0x4cb8160 -6414> 2014-05-19 18:36:27.070357 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:27.070162) v2 ==== 47+0+0 (1861970833 0 0) 0x55181c0 con 0x4cb82c0 -6413> 2014-05-19 18:36:27.070437 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6412> 2014-05-19 18:36:27.070540 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:27.070162) v2 -- ?+0 0x5c05880 con 0x4cb82c0 -6411> 2014-05-19 18:36:27.070581 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6410> 2014-05-19 18:36:27.340595 7fd03c6ae700 20 OSD::op_tp worker waiting -6409> 2014-05-19 18:36:27.340618 7fd041eb9700 20 OSD::op_tp worker waiting -6408> 2014-05-19 18:36:27.340654 7fd03e6b2700 20 OSD::op_tp worker waiting -6407> 2014-05-19 18:36:27.341078 7fd03d6b0700 20 OSD::op_tp worker waiting -6406> 2014-05-19 18:36:27.341190 7fd0436bc700 20 OSD::op_tp worker waiting -6405> 2014-05-19 18:36:27.341316 7fd03bead700 20 OSD::op_tp worker waiting -6404> 2014-05-19 18:36:27.341413 7fd03f6b4700 20 OSD::op_tp worker waiting -6403> 2014-05-19 18:36:27.355291 7fd0446be700 20 OSD::op_tp worker waiting -6402> 2014-05-19 18:36:27.355309 7fd03feb5700 20 OSD::op_tp worker waiting -6401> 2014-05-19 18:36:27.362331 7fd040eb7700 20 OSD::op_tp worker waiting -6400> 2014-05-19 18:36:27.367622 7fd0406b6700 20 OSD::op_tp worker waiting -6399> 2014-05-19 18:36:27.389127 7fd039ea9700 20 OSD::disk_tp worker waiting -6398> 2014-05-19 18:36:27.392778 7fd03aeab700 20 OSD::op_tp worker waiting -6397> 2014-05-19 18:36:27.392833 7fd03eeb3700 20 OSD::op_tp worker waiting -6396> 2014-05-19 18:36:27.410727 7fd042ebb700 20 OSD::op_tp worker waiting -6395> 2014-05-19 18:36:27.410749 7fd03deb1700 20 OSD::op_tp worker waiting -6394> 2014-05-19 18:36:27.419256 7fd04e104700 20 KeyValueStore::op_tp worker waiting -6393> 2014-05-19 18:36:27.419259 7fd03b6ac700 20 OSD::op_tp worker waiting -6392> 2014-05-19 18:36:27.419353 7fd0426ba700 20 OSD::op_tp worker waiting -6391> 2014-05-19 18:36:27.419376 7fd0416b8700 20 OSD::op_tp worker waiting -6390> 2014-05-19 18:36:27.419402 7fd03ceaf700 20 OSD::op_tp worker waiting -6389> 2014-05-19 18:36:27.489957 7fd04f907700 5 osd.0 21 tick -6388> 2014-05-19 18:36:27.489974 7fd04f907700 7 osd.0 21 do_mon_report -6387> 2014-05-19 18:36:27.489978 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -6386> 2014-05-19 18:36:27.489982 7fd04f907700 20 osd.0 21 send_pg_stats -6385> 2014-05-19 18:36:27.490008 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6384> 2014-05-19 18:36:27.490016 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes -6383> 2014-05-19 18:36:27.490029 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -6382> 2014-05-19 18:36:27.490032 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 230.214 < min (86400 seconds) -6381> 2014-05-19 18:36:27.490050 7fd04f907700 20 osd.0 21 sched_scrub done -6380> 2014-05-19 18:36:27.490053 7fd04f907700 10 osd.0 21 do_waiters -- start -6379> 2014-05-19 18:36:27.490055 7fd04f907700 10 osd.0 21 do_waiters -- finish -6378> 2014-05-19 18:36:27.989620 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:31.046489) v2 ==== 47+0+0 (349561710 0 0) 0x55c3880 con 0x61ae160 -6377> 2014-05-19 18:36:27.989671 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:31.046489) v2 -- ?+0 0x77e81c0 con 0x61ae160 -6376> 2014-05-19 18:36:27.989683 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:31.046489) v2 ==== 47+0+0 (349561710 0 0) 0x65436c0 con 0x61b0420 -6375> 2014-05-19 18:36:27.989754 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6374> 2014-05-19 18:36:27.989830 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:31.046489) v2 -- ?+0 0x55181c0 con 0x61b0420 -6373> 2014-05-19 18:36:27.989862 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6372> 2014-05-19 18:36:28.055205 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -6371> 2014-05-19 18:36:28.055223 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -6370> 2014-05-19 18:36:28.055231 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x5f0a380 con 0x3622580 -6369> 2014-05-19 18:36:28.055256 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x6378e00 con 0x3622840 -6368> 2014-05-19 18:36:28.055295 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x6d616c0 con 0x3622b00 -6367> 2014-05-19 18:36:28.055309 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x682fdc0 con 0x3622dc0 -6366> 2014-05-19 18:36:28.055323 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x5f7ac40 con 0x30594a0 -6365> 2014-05-19 18:36:28.055350 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x725d880 con 0x3622f20 -6364> 2014-05-19 18:36:28.055379 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x59bcfc0 con 0x5e7ef20 -6363> 2014-05-19 18:36:28.055406 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x60e0700 con 0x5e7f1e0 -6362> 2014-05-19 18:36:28.055439 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x5c05500 con 0x5e7e2c0 -6361> 2014-05-19 18:36:28.055466 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x60e2000 con 0x5d7bb80 -6360> 2014-05-19 18:36:28.055489 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x6c4c540 con 0x5e7ec60 -6359> 2014-05-19 18:36:28.055512 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x6008380 con 0x531b760 -6358> 2014-05-19 18:36:28.055537 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x51a6540 con 0x53bba20 -6357> 2014-05-19 18:36:28.055554 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x36f68c0 con 0x51cac60 -6356> 2014-05-19 18:36:28.055577 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x7122540 con 0x5bd66e0 -6355> 2014-05-19 18:36:28.055602 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:28.055229) v2 -- ?+0 0x74a76c0 con 0x5bd6840 -6354> 2014-05-19 18:36:28.055591 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x66f8000 con 0x3622580 -6353> 2014-05-19 18:36:28.055667 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6352> 2014-05-19 18:36:28.055715 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x5c041c0 con 0x3622dc0 -6351> 2014-05-19 18:36:28.055755 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6350> 2014-05-19 18:36:28.055779 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x59bddc0 con 0x3622b00 -6349> 2014-05-19 18:36:28.055815 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6348> 2014-05-19 18:36:28.055830 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x64ae000 con 0x5e7e2c0 -6347> 2014-05-19 18:36:28.055943 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6346> 2014-05-19 18:36:28.055979 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x6505500 con 0x30594a0 -6345> 2014-05-19 18:36:28.056006 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6344> 2014-05-19 18:36:28.056016 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x7930540 con 0x3622f20 -6343> 2014-05-19 18:36:28.056032 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6342> 2014-05-19 18:36:28.056046 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x73c2540 con 0x3622840 -6341> 2014-05-19 18:36:28.056063 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6340> 2014-05-19 18:36:28.056071 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x725da40 con 0x5e7ef20 -6339> 2014-05-19 18:36:28.056086 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6338> 2014-05-19 18:36:28.056093 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x77468c0 con 0x5e7f1e0 -6337> 2014-05-19 18:36:28.056108 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6336> 2014-05-19 18:36:28.056115 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x8389880 con 0x5d7bb80 -6335> 2014-05-19 18:36:28.056130 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6334> 2014-05-19 18:36:28.056140 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x6e2a700 con 0x5e7ec60 -6333> 2014-05-19 18:36:28.056157 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6332> 2014-05-19 18:36:28.056165 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x6c4c000 con 0x531b760 -6331> 2014-05-19 18:36:28.056180 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6330> 2014-05-19 18:36:28.056188 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x772ce00 con 0x53bba20 -6329> 2014-05-19 18:36:28.056206 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6328> 2014-05-19 18:36:28.056213 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x55c9a40 con 0x51cac60 -6327> 2014-05-19 18:36:28.056228 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6326> 2014-05-19 18:36:28.056236 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x649fa40 con 0x5bd66e0 -6325> 2014-05-19 18:36:28.056250 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6324> 2014-05-19 18:36:28.056258 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 68 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.055229) v2 ==== 47+0+0 (2739277196 0 0) 0x70ea380 con 0x5bd6840 -6323> 2014-05-19 18:36:28.056273 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6322> 2014-05-19 18:36:28.484463 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:28.484282) v2 ==== 47+0+0 (3484262981 0 0) 0x662b500 con 0x4cb8420 -6321> 2014-05-19 18:36:28.484518 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.484282) v2 -- ?+0 0x65436c0 con 0x4cb8420 -6320> 2014-05-19 18:36:28.484545 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6319> 2014-05-19 18:36:28.484469 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:28.484282) v2 ==== 47+0+0 (3484262981 0 0) 0x6504a80 con 0x4cb86e0 -6318> 2014-05-19 18:36:28.484642 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.484282) v2 -- ?+0 0x55c3880 con 0x4cb86e0 -6317> 2014-05-19 18:36:28.484678 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6316> 2014-05-19 18:36:28.490141 7fd04f907700 5 osd.0 21 tick -6315> 2014-05-19 18:36:28.490189 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes -6314> 2014-05-19 18:36:28.490202 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -6313> 2014-05-19 18:36:28.490210 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6312> 2014-05-19 18:36:28.490205 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 231.214 < min (86400 seconds) -6311> 2014-05-19 18:36:28.490223 7fd04f907700 20 osd.0 21 sched_scrub done -6310> 2014-05-19 18:36:28.490227 7fd04f907700 10 osd.0 21 do_waiters -- start -6309> 2014-05-19 18:36:28.490229 7fd04f907700 10 osd.0 21 do_waiters -- finish -6308> 2014-05-19 18:36:28.774161 7fd04e905700 20 KeyValueStore::op_tp worker waiting -6307> 2014-05-19 18:36:28.774175 7fd043ebd700 20 OSD::op_tp worker waiting -6306> 2014-05-19 18:36:28.871219 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:33.957263) v2 ==== 47+0+0 (588539321 0 0) 0x71236c0 con 0x31e1600 -6305> 2014-05-19 18:36:28.871232 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:33.957263) v2 ==== 47+0+0 (588539321 0 0) 0x75b0e00 con 0x61b0160 -6304> 2014-05-19 18:36:28.871268 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:33.957263) v2 -- ?+0 0x6504a80 con 0x31e1600 -6303> 2014-05-19 18:36:28.871306 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6302> 2014-05-19 18:36:28.871370 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:33.957263) v2 -- ?+0 0x662b500 con 0x61b0160 -6301> 2014-05-19 18:36:28.871411 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6300> 2014-05-19 18:36:28.946519 7fd0396a8700 20 OSD::command_tp worker waiting -6299> 2014-05-19 18:36:28.985066 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:28.984852) v2 ==== 47+0+0 (134403495 0 0) 0x79e7340 con 0x4cb8420 -6298> 2014-05-19 18:36:28.985119 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.984852) v2 -- ?+0 0x75b0e00 con 0x4cb8420 -6297> 2014-05-19 18:36:28.985014 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:28.984852) v2 ==== 47+0+0 (134403495 0 0) 0x59bc1c0 con 0x4cb86e0 -6296> 2014-05-19 18:36:28.985165 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6295> 2014-05-19 18:36:28.985199 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:28.984852) v2 -- ?+0 0x71236c0 con 0x4cb86e0 -6294> 2014-05-19 18:36:28.985232 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6293> 2014-05-19 18:36:29.340672 7fd03c6ae700 20 OSD::op_tp worker waiting -6292> 2014-05-19 18:36:29.340696 7fd041eb9700 20 OSD::op_tp worker waiting -6291> 2014-05-19 18:36:29.340714 7fd03e6b2700 20 OSD::op_tp worker waiting -6290> 2014-05-19 18:36:29.341150 7fd03d6b0700 20 OSD::op_tp worker waiting -6289> 2014-05-19 18:36:29.341259 7fd0436bc700 20 OSD::op_tp worker waiting -6288> 2014-05-19 18:36:29.341377 7fd03bead700 20 OSD::op_tp worker waiting -6287> 2014-05-19 18:36:29.341491 7fd03f6b4700 20 OSD::op_tp worker waiting -6286> 2014-05-19 18:36:29.355384 7fd0446be700 20 OSD::op_tp worker waiting -6285> 2014-05-19 18:36:29.355426 7fd03feb5700 20 OSD::op_tp worker waiting -6284> 2014-05-19 18:36:29.362408 7fd040eb7700 20 OSD::op_tp worker waiting -6283> 2014-05-19 18:36:29.367719 7fd0406b6700 20 OSD::op_tp worker waiting -6282> 2014-05-19 18:36:29.389224 7fd039ea9700 20 OSD::disk_tp worker waiting -6281> 2014-05-19 18:36:29.392867 7fd03aeab700 20 OSD::op_tp worker waiting -6280> 2014-05-19 18:36:29.392944 7fd03eeb3700 20 OSD::op_tp worker waiting -6279> 2014-05-19 18:36:29.410828 7fd042ebb700 20 OSD::op_tp worker waiting -6278> 2014-05-19 18:36:29.410888 7fd03deb1700 20 OSD::op_tp worker waiting -6277> 2014-05-19 18:36:29.419354 7fd04e104700 20 KeyValueStore::op_tp worker waiting -6276> 2014-05-19 18:36:29.419355 7fd03b6ac700 20 OSD::op_tp worker waiting -6275> 2014-05-19 18:36:29.419439 7fd0426ba700 20 OSD::op_tp worker waiting -6274> 2014-05-19 18:36:29.419465 7fd0416b8700 20 OSD::op_tp worker waiting -6273> 2014-05-19 18:36:29.419495 7fd03ceaf700 20 OSD::op_tp worker waiting -6272> 2014-05-19 18:36:29.490315 7fd04f907700 5 osd.0 21 tick -6271> 2014-05-19 18:36:29.490367 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6270> 2014-05-19 18:36:29.490363 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes -6269> 2014-05-19 18:36:29.490376 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -6268> 2014-05-19 18:36:29.490379 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 232.214 < min (86400 seconds) -6267> 2014-05-19 18:36:29.490398 7fd04f907700 20 osd.0 21 sched_scrub done -6266> 2014-05-19 18:36:29.490401 7fd04f907700 10 osd.0 21 do_waiters -- start -6265> 2014-05-19 18:36:29.490402 7fd04f907700 10 osd.0 21 do_waiters -- finish -6264> 2014-05-19 18:36:29.558932 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:32.615794) v2 ==== 47+0+0 (4225162501 0 0) 0x70a9340 con 0x4cb9340 -6263> 2014-05-19 18:36:29.558964 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:32.615794) v2 ==== 47+0+0 (4225162501 0 0) 0x6aa56c0 con 0x4cb8dc0 -6262> 2014-05-19 18:36:29.558985 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.615794) v2 -- ?+0 0x79e7340 con 0x4cb9340 -6261> 2014-05-19 18:36:29.559090 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6260> 2014-05-19 18:36:29.559119 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.615794) v2 -- ?+0 0x59bc1c0 con 0x4cb8dc0 -6259> 2014-05-19 18:36:29.559155 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6258> 2014-05-19 18:36:29.821535 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:32.878373) v2 ==== 47+0+0 (3951532010 0 0) 0x798e700 con 0x61ae420 -6257> 2014-05-19 18:36:29.821541 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:32.878373) v2 ==== 47+0+0 (3951532010 0 0) 0x6e188c0 con 0x618fe40 -6256> 2014-05-19 18:36:29.821594 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.878373) v2 -- ?+0 0x6aa56c0 con 0x61ae420 -6255> 2014-05-19 18:36:29.821638 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6254> 2014-05-19 18:36:29.821663 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.878373) v2 -- ?+0 0x70a9340 con 0x618fe40 -6253> 2014-05-19 18:36:29.821691 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6252> 2014-05-19 18:36:29.907127 7fd0456c0700 10 monclient: tick -6251> 2014-05-19 18:36:29.907139 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:35:59.907138) -6250> 2014-05-19 18:36:29.907157 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:36:29.907156; renew after: 2014-05-19 18:38:19.906664) -- no -6249> 2014-05-19 18:36:29.970889 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:29.970707) v2 ==== 47+0+0 (3623132993 0 0) 0x5519880 con 0x4cb82c0 -6248> 2014-05-19 18:36:29.970951 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:29.970707) v2 -- ?+0 0x6e188c0 con 0x4cb82c0 -6247> 2014-05-19 18:36:29.970986 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6246> 2014-05-19 18:36:29.971023 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:29.970707) v2 ==== 47+0+0 (3623132993 0 0) 0x5f06540 con 0x4cb8160 -6245> 2014-05-19 18:36:29.971066 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:29.970707) v2 -- ?+0 0x798e700 con 0x4cb8160 -6244> 2014-05-19 18:36:29.971105 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6243> 2014-05-19 18:36:30.059453 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:33.116310) v2 ==== 47+0+0 (653286048 0 0) 0x691b340 con 0x4cb9340 -6242> 2014-05-19 18:36:30.059491 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:33.116310) v2 -- ?+0 0x5519880 con 0x4cb9340 -6241> 2014-05-19 18:36:30.059477 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:33.116310) v2 ==== 47+0+0 (653286048 0 0) 0x6aa4700 con 0x4cb8dc0 -6240> 2014-05-19 18:36:30.059524 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6239> 2014-05-19 18:36:30.059561 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:33.116310) v2 -- ?+0 0x5f06540 con 0x4cb8dc0 -6238> 2014-05-19 18:36:30.059597 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6237> 2014-05-19 18:36:30.386349 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:35.472462) v2 ==== 47+0+0 (2462226948 0 0) 0x6710e00 con 0x5b21600 -6236> 2014-05-19 18:36:30.386387 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:35.472462) v2 ==== 47+0+0 (2462226948 0 0) 0x3438fc0 con 0x5f5b760 -6235> 2014-05-19 18:36:30.386393 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:35.472462) v2 -- ?+0 0x6aa4700 con 0x5b21600 -6234> 2014-05-19 18:36:30.386448 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6233> 2014-05-19 18:36:30.386469 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:35.472462) v2 -- ?+0 0x691b340 con 0x5f5b760 -6232> 2014-05-19 18:36:30.386514 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6231> 2014-05-19 18:36:30.490486 7fd04f907700 5 osd.0 21 tick -6230> 2014-05-19 18:36:30.490506 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -6229> 2014-05-19 18:36:30.490511 7fd04f907700 10 osd.0 21 do_waiters -- start -6228> 2014-05-19 18:36:30.490513 7fd04f907700 10 osd.0 21 do_waiters -- finish -6227> 2014-05-19 18:36:30.490542 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6226> 2014-05-19 18:36:30.774258 7fd04e905700 20 KeyValueStore::op_tp worker waiting -6225> 2014-05-19 18:36:30.774260 7fd043ebd700 20 OSD::op_tp worker waiting -6224> 2014-05-19 18:36:30.946612 7fd0396a8700 20 OSD::command_tp worker waiting -6223> 2014-05-19 18:36:31.311202 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:36.397312) v2 ==== 47+0+0 (1240713438 0 0) 0x60e2a80 con 0x30946e0 -6222> 2014-05-19 18:36:31.311212 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 63 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:36.397312) v2 ==== 47+0+0 (1240713438 0 0) 0x798fc00 con 0x3094160 -6221> 2014-05-19 18:36:31.311257 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.397312) v2 -- ?+0 0x6710e00 con 0x30946e0 -6220> 2014-05-19 18:36:31.311288 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6219> 2014-05-19 18:36:31.311304 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.397312) v2 -- ?+0 0x3438fc0 con 0x3094160 -6218> 2014-05-19 18:36:31.311332 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6217> 2014-05-19 18:36:31.340755 7fd03c6ae700 20 OSD::op_tp worker waiting -6216> 2014-05-19 18:36:31.340774 7fd041eb9700 20 OSD::op_tp worker waiting -6215> 2014-05-19 18:36:31.340802 7fd03e6b2700 20 OSD::op_tp worker waiting -6214> 2014-05-19 18:36:31.341218 7fd03d6b0700 20 OSD::op_tp worker waiting -6213> 2014-05-19 18:36:31.341339 7fd0436bc700 20 OSD::op_tp worker waiting -6212> 2014-05-19 18:36:31.341448 7fd03bead700 20 OSD::op_tp worker waiting -6211> 2014-05-19 18:36:31.341587 7fd03f6b4700 20 OSD::op_tp worker waiting -6210> 2014-05-19 18:36:31.355481 7fd0446be700 20 OSD::op_tp worker waiting -6209> 2014-05-19 18:36:31.355502 7fd03feb5700 20 OSD::op_tp worker waiting -6208> 2014-05-19 18:36:31.362502 7fd040eb7700 20 OSD::op_tp worker waiting -6207> 2014-05-19 18:36:31.367823 7fd0406b6700 20 OSD::op_tp worker waiting -6206> 2014-05-19 18:36:31.389319 7fd039ea9700 20 OSD::disk_tp worker waiting -6205> 2014-05-19 18:36:31.392948 7fd03aeab700 20 OSD::op_tp worker waiting -6204> 2014-05-19 18:36:31.393030 7fd03eeb3700 20 OSD::op_tp worker waiting -6203> 2014-05-19 18:36:31.410926 7fd042ebb700 20 OSD::op_tp worker waiting -6202> 2014-05-19 18:36:31.410968 7fd03deb1700 20 OSD::op_tp worker waiting -6201> 2014-05-19 18:36:31.419461 7fd03b6ac700 20 OSD::op_tp worker waiting -6200> 2014-05-19 18:36:31.419460 7fd04e104700 20 KeyValueStore::op_tp worker waiting -6199> 2014-05-19 18:36:31.419519 7fd0426ba700 20 OSD::op_tp worker waiting -6198> 2014-05-19 18:36:31.419541 7fd0416b8700 20 OSD::op_tp worker waiting -6197> 2014-05-19 18:36:31.419566 7fd03ceaf700 20 OSD::op_tp worker waiting -6196> 2014-05-19 18:36:31.490161 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:34.547020) v2 ==== 47+0+0 (789580583 0 0) 0x55c28c0 con 0x61ae160 -6195> 2014-05-19 18:36:31.490162 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:34.547020) v2 ==== 47+0+0 (789580583 0 0) 0x65421c0 con 0x61b0420 -6194> 2014-05-19 18:36:31.490215 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.547020) v2 -- ?+0 0x60e2a80 con 0x61ae160 -6193> 2014-05-19 18:36:31.490251 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6192> 2014-05-19 18:36:31.490277 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.547020) v2 -- ?+0 0x798fc00 con 0x61b0420 -6191> 2014-05-19 18:36:31.490307 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6190> 2014-05-19 18:36:31.490571 7fd04f907700 5 osd.0 21 tick -6189> 2014-05-19 18:36:31.490627 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6188> 2014-05-19 18:36:31.490623 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.07 < max 0.5 = yes -6187> 2014-05-19 18:36:31.490651 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -6186> 2014-05-19 18:36:31.490657 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 234.215 < min (86400 seconds) -6185> 2014-05-19 18:36:31.490682 7fd04f907700 20 osd.0 21 sched_scrub done -6184> 2014-05-19 18:36:31.490687 7fd04f907700 10 osd.0 21 do_waiters -- start -6183> 2014-05-19 18:36:31.490691 7fd04f907700 10 osd.0 21 do_waiters -- finish -6182> 2014-05-19 18:36:31.885649 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:31.885481) v2 ==== 47+0+0 (2640082069 0 0) 0x55f56c0 con 0x4cb86e0 -6181> 2014-05-19 18:36:31.885701 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:31.885481) v2 -- ?+0 0x55c28c0 con 0x4cb86e0 -6180> 2014-05-19 18:36:31.885740 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6179> 2014-05-19 18:36:31.885711 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:31.885481) v2 ==== 47+0+0 (2640082069 0 0) 0x5e29340 con 0x4cb8420 -6178> 2014-05-19 18:36:31.885758 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:31.885481) v2 -- ?+0 0x65421c0 con 0x4cb8420 -6177> 2014-05-19 18:36:31.885836 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6176> 2014-05-19 18:36:31.990602 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:35.047516) v2 ==== 47+0+0 (3927650715 0 0) 0x55c3500 con 0x61ae160 -6175> 2014-05-19 18:36:31.990644 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:35.047516) v2 -- ?+0 0x55f56c0 con 0x61ae160 -6174> 2014-05-19 18:36:31.990666 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:35.047516) v2 ==== 47+0+0 (3927650715 0 0) 0x6543180 con 0x61b0420 -6173> 2014-05-19 18:36:31.990736 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6172> 2014-05-19 18:36:31.990781 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:35.047516) v2 -- ?+0 0x5e29340 con 0x61b0420 -6171> 2014-05-19 18:36:31.990820 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6170> 2014-05-19 18:36:32.371566 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:37.457773) v2 ==== 47+0+0 (2222634050 0 0) 0x71cafc0 con 0x61b0160 -6169> 2014-05-19 18:36:32.371569 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:37.457773) v2 ==== 47+0+0 (2222634050 0 0) 0x79e68c0 con 0x31e1600 -6168> 2014-05-19 18:36:32.371626 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:37.457773) v2 -- ?+0 0x6543180 con 0x61b0160 -6167> 2014-05-19 18:36:32.371660 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6166> 2014-05-19 18:36:32.371707 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:37.457773) v2 -- ?+0 0x55c3500 con 0x31e1600 -6165> 2014-05-19 18:36:32.371751 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6164> 2014-05-19 18:36:32.490773 7fd04f907700 5 osd.0 21 tick -6163> 2014-05-19 18:36:32.490791 7fd04f907700 7 osd.0 21 do_mon_report -6162> 2014-05-19 18:36:32.490796 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -6161> 2014-05-19 18:36:32.490802 7fd04f907700 20 osd.0 21 send_pg_stats -6160> 2014-05-19 18:36:32.490813 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -6159> 2014-05-19 18:36:32.490819 7fd04f907700 10 osd.0 21 do_waiters -- start -6158> 2014-05-19 18:36:32.490823 7fd04f907700 10 osd.0 21 do_waiters -- finish -6157> 2014-05-19 18:36:32.490851 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6156> 2014-05-19 18:36:32.755751 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -6155> 2014-05-19 18:36:32.755769 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -6154> 2014-05-19 18:36:32.755777 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x5667c00 con 0x3622580 -6153> 2014-05-19 18:36:32.755803 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x7971dc0 con 0x3622840 -6152> 2014-05-19 18:36:32.755840 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x6b71c00 con 0x3622b00 -6151> 2014-05-19 18:36:32.755855 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x77461c0 con 0x3622dc0 -6150> 2014-05-19 18:36:32.755884 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x5f7b340 con 0x30594a0 -6149> 2014-05-19 18:36:32.755913 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x511ddc0 con 0x3622f20 -6148> 2014-05-19 18:36:32.756040 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x72e41c0 con 0x5e7ef20 -6147> 2014-05-19 18:36:32.756061 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x68ab500 con 0x5e7f1e0 -6146> 2014-05-19 18:36:32.756077 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x5c05340 con 0x5e7e2c0 -6145> 2014-05-19 18:36:32.756091 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x669e540 con 0x5d7bb80 -6144> 2014-05-19 18:36:32.756105 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x5496000 con 0x5e7ec60 -6143> 2014-05-19 18:36:32.756119 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x6009c00 con 0x531b760 -6142> 2014-05-19 18:36:32.756132 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x36f7340 con 0x53bba20 -6141> 2014-05-19 18:36:32.756147 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x7122700 con 0x51cac60 -6140> 2014-05-19 18:36:32.756172 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x7a5e1c0 con 0x5bd66e0 -6139> 2014-05-19 18:36:32.756198 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:32.755776) v2 -- ?+0 0x55f4fc0 con 0x5bd6840 -6138> 2014-05-19 18:36:32.756195 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x66f9500 con 0x3622580 -6137> 2014-05-19 18:36:32.756233 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6136> 2014-05-19 18:36:32.756248 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x73c28c0 con 0x3622840 -6135> 2014-05-19 18:36:32.756313 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6134> 2014-05-19 18:36:32.756335 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x59bcc40 con 0x3622b00 -6133> 2014-05-19 18:36:32.756357 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6132> 2014-05-19 18:36:32.756370 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x60a0540 con 0x3622dc0 -6131> 2014-05-19 18:36:32.756415 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6130> 2014-05-19 18:36:32.756430 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x5f328c0 con 0x30594a0 -6129> 2014-05-19 18:36:32.756454 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6128> 2014-05-19 18:36:32.756525 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x7931a40 con 0x3622f20 -6127> 2014-05-19 18:36:32.756551 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6126> 2014-05-19 18:36:32.756604 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x7747180 con 0x5e7f1e0 -6125> 2014-05-19 18:36:32.756634 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6124> 2014-05-19 18:36:32.756648 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x64afc00 con 0x5e7e2c0 -6123> 2014-05-19 18:36:32.756677 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6122> 2014-05-19 18:36:32.756687 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x3510380 con 0x5d7bb80 -6121> 2014-05-19 18:36:32.756704 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6120> 2014-05-19 18:36:32.756710 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x6e2ba40 con 0x5e7ec60 -6119> 2014-05-19 18:36:32.756724 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6118> 2014-05-19 18:36:32.756732 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x6008e00 con 0x531b760 -6117> 2014-05-19 18:36:32.756749 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6116> 2014-05-19 18:36:32.756757 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x772cc40 con 0x53bba20 -6115> 2014-05-19 18:36:32.756784 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6114> 2014-05-19 18:36:32.756795 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x55c8700 con 0x51cac60 -6113> 2014-05-19 18:36:32.756813 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6112> 2014-05-19 18:36:32.756820 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x784d6c0 con 0x5bd66e0 -6111> 2014-05-19 18:36:32.756834 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6110> 2014-05-19 18:36:32.756842 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 69 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x70ea700 con 0x5bd6840 -6109> 2014-05-19 18:36:32.756856 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6108> 2014-05-19 18:36:32.756864 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:32.755776) v2 ==== 47+0+0 (2295230210 0 0) 0x7a5e8c0 con 0x5e7ef20 -6107> 2014-05-19 18:36:32.756879 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6106> 2014-05-19 18:36:32.774342 7fd043ebd700 20 OSD::op_tp worker waiting -6105> 2014-05-19 18:36:32.774347 7fd04e905700 20 KeyValueStore::op_tp worker waiting -6104> 2014-05-19 18:36:32.872140 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:37.958384) v2 ==== 47+0+0 (3306215810 0 0) 0x71cb340 con 0x61b0160 -6103> 2014-05-19 18:36:32.872209 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:37.958384) v2 -- ?+0 0x71cafc0 con 0x61b0160 -6102> 2014-05-19 18:36:32.872144 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:37.958384) v2 ==== 47+0+0 (3306215810 0 0) 0x79e6c40 con 0x31e1600 -6101> 2014-05-19 18:36:32.872302 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6100> 2014-05-19 18:36:32.872324 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:37.958384) v2 -- ?+0 0x79e68c0 con 0x31e1600 -6099> 2014-05-19 18:36:32.872359 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6098> 2014-05-19 18:36:32.946705 7fd0396a8700 20 OSD::command_tp worker waiting -6097> 2014-05-19 18:36:32.959930 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:36.016840) v2 ==== 47+0+0 (3374648556 0 0) 0x6aa5880 con 0x4cb8dc0 -6096> 2014-05-19 18:36:32.959943 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:36.016840) v2 ==== 47+0+0 (3374648556 0 0) 0x55c88c0 con 0x4cb9340 -6095> 2014-05-19 18:36:32.959980 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.016840) v2 -- ?+0 0x79e6c40 con 0x4cb8dc0 -6094> 2014-05-19 18:36:32.960014 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6093> 2014-05-19 18:36:32.960034 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.016840) v2 -- ?+0 0x71cb340 con 0x4cb9340 -6092> 2014-05-19 18:36:32.960076 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6091> 2014-05-19 18:36:33.340833 7fd03c6ae700 20 OSD::op_tp worker waiting -6090> 2014-05-19 18:36:33.340852 7fd041eb9700 20 OSD::op_tp worker waiting -6089> 2014-05-19 18:36:33.340896 7fd03e6b2700 20 OSD::op_tp worker waiting -6088> 2014-05-19 18:36:33.341287 7fd03d6b0700 20 OSD::op_tp worker waiting -6087> 2014-05-19 18:36:33.341406 7fd0436bc700 20 OSD::op_tp worker waiting -6086> 2014-05-19 18:36:33.341516 7fd03bead700 20 OSD::op_tp worker waiting -6085> 2014-05-19 18:36:33.341666 7fd03f6b4700 20 OSD::op_tp worker waiting -6084> 2014-05-19 18:36:33.355581 7fd0446be700 20 OSD::op_tp worker waiting -6083> 2014-05-19 18:36:33.355603 7fd03feb5700 20 OSD::op_tp worker waiting -6082> 2014-05-19 18:36:33.362589 7fd040eb7700 20 OSD::op_tp worker waiting -6081> 2014-05-19 18:36:33.367919 7fd0406b6700 20 OSD::op_tp worker waiting -6080> 2014-05-19 18:36:33.389414 7fd039ea9700 20 OSD::disk_tp worker waiting -6079> 2014-05-19 18:36:33.393029 7fd03aeab700 20 OSD::op_tp worker waiting -6078> 2014-05-19 18:36:33.393112 7fd03eeb3700 20 OSD::op_tp worker waiting -6077> 2014-05-19 18:36:33.411010 7fd042ebb700 20 OSD::op_tp worker waiting -6076> 2014-05-19 18:36:33.411048 7fd03deb1700 20 OSD::op_tp worker waiting -6075> 2014-05-19 18:36:33.419567 7fd04e104700 20 KeyValueStore::op_tp worker waiting -6074> 2014-05-19 18:36:33.419572 7fd03b6ac700 20 OSD::op_tp worker waiting -6073> 2014-05-19 18:36:33.419598 7fd0426ba700 20 OSD::op_tp worker waiting -6072> 2014-05-19 18:36:33.419620 7fd0416b8700 20 OSD::op_tp worker waiting -6071> 2014-05-19 18:36:33.419644 7fd03ceaf700 20 OSD::op_tp worker waiting -6070> 2014-05-19 18:36:33.490893 7fd04f907700 5 osd.0 21 tick -6069> 2014-05-19 18:36:33.490914 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -6068> 2014-05-19 18:36:33.490919 7fd04f907700 10 osd.0 21 do_waiters -- start -6067> 2014-05-19 18:36:33.490921 7fd04f907700 10 osd.0 21 do_waiters -- finish -6066> 2014-05-19 18:36:33.490944 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6065> 2014-05-19 18:36:34.456326 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -6064> 2014-05-19 18:36:34.456344 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -6063> 2014-05-19 18:36:34.456352 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x6542000 con 0x3622580 -6062> 2014-05-19 18:36:34.456377 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x5f06000 con 0x3622840 -6061> 2014-05-19 18:36:34.456414 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x6711180 con 0x3622b00 -6060> 2014-05-19 18:36:34.456429 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x7733340 con 0x3622dc0 -6059> 2014-05-19 18:36:34.456443 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x7930540 con 0x30594a0 -6058> 2014-05-19 18:36:34.456470 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x6505500 con 0x3622f20 -6057> 2014-05-19 18:36:34.456496 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x64ae000 con 0x5e7ef20 -6056> 2014-05-19 18:36:34.456522 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x59bddc0 con 0x5e7f1e0 -6055> 2014-05-19 18:36:34.456547 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x5c041c0 con 0x5e7e2c0 -6054> 2014-05-19 18:36:34.456572 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x66f8000 con 0x5d7bb80 -6053> 2014-05-19 18:36:34.456630 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x6c4cc40 con 0x5e7ec60 -6052> 2014-05-19 18:36:34.456657 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x725ca80 con 0x531b760 -6051> 2014-05-19 18:36:34.456678 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x6113180 con 0x53bba20 -6050> 2014-05-19 18:36:34.456700 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x6504e00 con 0x51cac60 -6049> 2014-05-19 18:36:34.456726 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x70ea540 con 0x5bd66e0 -6048> 2014-05-19 18:36:34.456752 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:34.456350) v2 -- ?+0 0x649e540 con 0x5bd6840 -6047> 2014-05-19 18:36:34.456753 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x5e288c0 con 0x3622b00 -6046> 2014-05-19 18:36:34.456849 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6045> 2014-05-19 18:36:34.456872 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x66f8380 con 0x3622580 -6044> 2014-05-19 18:36:34.456897 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6043> 2014-05-19 18:36:34.456909 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x73c21c0 con 0x3622840 -6042> 2014-05-19 18:36:34.456923 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -6041> 2014-05-19 18:36:34.456932 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x60a1a40 con 0x3622dc0 -6040> 2014-05-19 18:36:34.456960 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -6039> 2014-05-19 18:36:34.456995 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x7a5e000 con 0x5e7ef20 -6038> 2014-05-19 18:36:34.457024 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6037> 2014-05-19 18:36:34.457031 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x6fe61c0 con 0x5e7e2c0 -6036> 2014-05-19 18:36:34.457046 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6035> 2014-05-19 18:36:34.457080 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x3511500 con 0x5d7bb80 -6034> 2014-05-19 18:36:34.457116 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -6033> 2014-05-19 18:36:34.457168 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x5f32380 con 0x30594a0 -6032> 2014-05-19 18:36:34.457193 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6031> 2014-05-19 18:36:34.457200 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x72e48c0 con 0x3622f20 -6030> 2014-05-19 18:36:34.457214 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -6029> 2014-05-19 18:36:34.457220 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x77476c0 con 0x5e7f1e0 -6028> 2014-05-19 18:36:34.457239 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6027> 2014-05-19 18:36:34.457247 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x6c4a700 con 0x5e7ec60 -6026> 2014-05-19 18:36:34.457262 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6025> 2014-05-19 18:36:34.457267 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x60096c0 con 0x531b760 -6024> 2014-05-19 18:36:34.457281 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -6023> 2014-05-19 18:36:34.457287 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x70cc1c0 con 0x53bba20 -6022> 2014-05-19 18:36:34.457301 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6021> 2014-05-19 18:36:34.457306 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x6543500 con 0x51cac60 -6020> 2014-05-19 18:36:34.457320 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -6019> 2014-05-19 18:36:34.457326 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x784ca80 con 0x5bd66e0 -6018> 2014-05-19 18:36:34.457341 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6017> 2014-05-19 18:36:34.457350 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 70 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.456350) v2 ==== 47+0+0 (1502428515 0 0) 0x70ebdc0 con 0x5bd6840 -6016> 2014-05-19 18:36:34.457364 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -6015> 2014-05-19 18:36:34.490992 7fd04f907700 5 osd.0 21 tick -6014> 2014-05-19 18:36:34.491053 7fd03a6aa700 20 OSD::recovery_tp worker waiting -6013> 2014-05-19 18:36:34.491049 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.06 < max 0.5 = yes -6012> 2014-05-19 18:36:34.491080 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -6011> 2014-05-19 18:36:34.491086 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 237.215 < min (86400 seconds) -6010> 2014-05-19 18:36:34.491111 7fd04f907700 20 osd.0 21 sched_scrub done -6009> 2014-05-19 18:36:34.491117 7fd04f907700 10 osd.0 21 do_waiters -- start -6008> 2014-05-19 18:36:34.491120 7fd04f907700 10 osd.0 21 do_waiters -- finish -6007> 2014-05-19 18:36:34.522018 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:37.578915) v2 ==== 47+0+0 (1890869772 0 0) 0x68aaa80 con 0x618fe40 -6006> 2014-05-19 18:36:34.522056 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:37.578915) v2 ==== 47+0+0 (1890869772 0 0) 0x5dfe8c0 con 0x61ae420 -6005> 2014-05-19 18:36:34.522069 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:37.578915) v2 -- ?+0 0x55c88c0 con 0x618fe40 -6004> 2014-05-19 18:36:34.522119 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6003> 2014-05-19 18:36:34.522157 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:37.578915) v2 -- ?+0 0x6aa5880 con 0x61ae420 -6002> 2014-05-19 18:36:34.522196 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -6001> 2014-05-19 18:36:34.671373 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:34.671207) v2 ==== 47+0+0 (1834226521 0 0) 0x5518700 con 0x4cb82c0 -6000> 2014-05-19 18:36:34.671405 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:34.671207) v2 ==== 47+0+0 (1834226521 0 0) 0x5f07340 con 0x4cb8160 -5999> 2014-05-19 18:36:34.671412 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.671207) v2 -- ?+0 0x68aaa80 con 0x4cb82c0 -5998> 2014-05-19 18:36:34.671568 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5997> 2014-05-19 18:36:34.671633 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:34.671207) v2 -- ?+0 0x5dfe8c0 con 0x4cb8160 -5996> 2014-05-19 18:36:34.671667 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5995> 2014-05-19 18:36:34.774434 7fd04e905700 20 KeyValueStore::op_tp worker waiting -5994> 2014-05-19 18:36:34.774437 7fd043ebd700 20 OSD::op_tp worker waiting -5993> 2014-05-19 18:36:34.891183 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:37.948113) v2 ==== 47+0+0 (701909257 0 0) 0x55c21c0 con 0x61ae160 -5992> 2014-05-19 18:36:34.891225 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:37.948113) v2 ==== 47+0+0 (701909257 0 0) 0x6543880 con 0x61b0420 -5991> 2014-05-19 18:36:34.891260 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:37.948113) v2 -- ?+0 0x5518700 con 0x61b0420 -5990> 2014-05-19 18:36:34.891296 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5989> 2014-05-19 18:36:34.891326 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:37.948113) v2 -- ?+0 0x5f07340 con 0x61ae160 -5988> 2014-05-19 18:36:34.891364 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5987> 2014-05-19 18:36:34.946798 7fd0396a8700 20 OSD::command_tp worker waiting -5986> 2014-05-19 18:36:35.340930 7fd041eb9700 20 OSD::op_tp worker waiting -5985> 2014-05-19 18:36:35.340959 7fd03e6b2700 20 OSD::op_tp worker waiting -5984> 2014-05-19 18:36:35.340981 7fd03c6ae700 20 OSD::op_tp worker waiting -5983> 2014-05-19 18:36:35.341356 7fd03d6b0700 20 OSD::op_tp worker waiting -5982> 2014-05-19 18:36:35.341476 7fd0436bc700 20 OSD::op_tp worker waiting -5981> 2014-05-19 18:36:35.341582 7fd03bead700 20 OSD::op_tp worker waiting -5980> 2014-05-19 18:36:35.341751 7fd03f6b4700 20 OSD::op_tp worker waiting -5979> 2014-05-19 18:36:35.355681 7fd0446be700 20 OSD::op_tp worker waiting -5978> 2014-05-19 18:36:35.355703 7fd03feb5700 20 OSD::op_tp worker waiting -5977> 2014-05-19 18:36:35.362682 7fd040eb7700 20 OSD::op_tp worker waiting -5976> 2014-05-19 18:36:35.368015 7fd0406b6700 20 OSD::op_tp worker waiting -5975> 2014-05-19 18:36:35.389508 7fd039ea9700 20 OSD::disk_tp worker waiting -5974> 2014-05-19 18:36:35.393110 7fd03aeab700 20 OSD::op_tp worker waiting -5973> 2014-05-19 18:36:35.393198 7fd03eeb3700 20 OSD::op_tp worker waiting -5972> 2014-05-19 18:36:35.411107 7fd042ebb700 20 OSD::op_tp worker waiting -5971> 2014-05-19 18:36:35.411127 7fd03deb1700 20 OSD::op_tp worker waiting -5970> 2014-05-19 18:36:35.419667 7fd04e104700 20 KeyValueStore::op_tp worker waiting -5969> 2014-05-19 18:36:35.419672 7fd03b6ac700 20 OSD::op_tp worker waiting -5968> 2014-05-19 18:36:35.419694 7fd0426ba700 20 OSD::op_tp worker waiting -5967> 2014-05-19 18:36:35.419717 7fd0416b8700 20 OSD::op_tp worker waiting -5966> 2014-05-19 18:36:35.419770 7fd03ceaf700 20 OSD::op_tp worker waiting -5965> 2014-05-19 18:36:35.491190 7fd04f907700 5 osd.0 21 tick -5964> 2014-05-19 18:36:35.491241 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5963> 2014-05-19 18:36:35.491238 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.06 < max 0.5 = yes -5962> 2014-05-19 18:36:35.491252 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -5961> 2014-05-19 18:36:35.491255 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 238.215 < min (86400 seconds) -5960> 2014-05-19 18:36:35.491273 7fd04f907700 20 osd.0 21 sched_scrub done -5959> 2014-05-19 18:36:35.491276 7fd04f907700 10 osd.0 21 do_waiters -- start -5958> 2014-05-19 18:36:35.491277 7fd04f907700 10 osd.0 21 do_waiters -- finish -5957> 2014-05-19 18:36:35.686662 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:40.772956) v2 ==== 47+0+0 (106237402 0 0) 0x70ebc00 con 0x5f5b760 -5956> 2014-05-19 18:36:35.686709 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.772956) v2 -- ?+0 0x6543880 con 0x5f5b760 -5955> 2014-05-19 18:36:35.686737 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5954> 2014-05-19 18:36:35.686778 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:40.772956) v2 ==== 47+0+0 (106237402 0 0) 0x5b281c0 con 0x5b21600 -5953> 2014-05-19 18:36:35.686827 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.772956) v2 -- ?+0 0x55c21c0 con 0x5b21600 -5952> 2014-05-19 18:36:35.686864 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5951> 2014-05-19 18:36:35.772621 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:40.858907) v2 ==== 47+0+0 (240002601 0 0) 0x662bc00 con 0x31e1600 -5950> 2014-05-19 18:36:35.772679 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.858907) v2 -- ?+0 0x5b281c0 con 0x31e1600 -5949> 2014-05-19 18:36:35.772713 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5948> 2014-05-19 18:36:35.772634 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:40.858907) v2 ==== 47+0+0 (240002601 0 0) 0x71cb180 con 0x61b0160 -5947> 2014-05-19 18:36:35.772835 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.858907) v2 -- ?+0 0x70ebc00 con 0x61b0160 -5946> 2014-05-19 18:36:35.772867 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5945> 2014-05-19 18:36:36.156872 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -5944> 2014-05-19 18:36:36.156890 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -5943> 2014-05-19 18:36:36.156898 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x2ee0fc0 con 0x3622580 -5942> 2014-05-19 18:36:36.156923 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x772d180 con 0x3622840 -5941> 2014-05-19 18:36:36.156961 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x7971500 con 0x3622b00 -5940> 2014-05-19 18:36:36.156975 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x8388540 con 0x3622dc0 -5939> 2014-05-19 18:36:36.156989 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x5f7a8c0 con 0x30594a0 -5938> 2014-05-19 18:36:36.157015 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x7746c40 con 0x3622f20 -5937> 2014-05-19 18:36:36.157049 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x5c05a40 con 0x5e7ef20 -5936> 2014-05-19 18:36:36.157078 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x60e01c0 con 0x5e7f1e0 -5935> 2014-05-19 18:36:36.157103 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x66f9dc0 con 0x5e7e2c0 -5934> 2014-05-19 18:36:36.157125 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x59bd500 con 0x5d7bb80 -5933> 2014-05-19 18:36:36.157149 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x7747880 con 0x5e7ec60 -5932> 2014-05-19 18:36:36.157173 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x6113880 con 0x531b760 -5931> 2014-05-19 18:36:36.157198 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x6504700 con 0x53bba20 -5930> 2014-05-19 18:36:36.157222 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x5cbc380 con 0x51cac60 -5929> 2014-05-19 18:36:36.157247 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x7122c40 con 0x5bd66e0 -5928> 2014-05-19 18:36:36.157270 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:36.156896) v2 -- ?+0 0x5dfee00 con 0x5bd6840 -5927> 2014-05-19 18:36:36.157259 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x66f88c0 con 0x3622580 -5926> 2014-05-19 18:36:36.157378 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5925> 2014-05-19 18:36:36.157393 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x5e28c40 con 0x3622b00 -5924> 2014-05-19 18:36:36.157413 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5923> 2014-05-19 18:36:36.157436 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x60a0700 con 0x3622dc0 -5922> 2014-05-19 18:36:36.157455 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5921> 2014-05-19 18:36:36.157463 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x6fe7880 con 0x5e7e2c0 -5920> 2014-05-19 18:36:36.157578 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5919> 2014-05-19 18:36:36.157595 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x72e4fc0 con 0x3622f20 -5918> 2014-05-19 18:36:36.157621 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5917> 2014-05-19 18:36:36.157640 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x73c3a40 con 0x3622840 -5916> 2014-05-19 18:36:36.157661 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5915> 2014-05-19 18:36:36.157670 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x7a5ea80 con 0x5e7ef20 -5914> 2014-05-19 18:36:36.157684 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5913> 2014-05-19 18:36:36.157690 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x79988c0 con 0x5e7f1e0 -5912> 2014-05-19 18:36:36.157704 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5911> 2014-05-19 18:36:36.157709 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x3511180 con 0x5d7bb80 -5910> 2014-05-19 18:36:36.157723 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5909> 2014-05-19 18:36:36.157729 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x6c4a000 con 0x5e7ec60 -5908> 2014-05-19 18:36:36.157742 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5907> 2014-05-19 18:36:36.157750 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x6009dc0 con 0x531b760 -5906> 2014-05-19 18:36:36.157767 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5905> 2014-05-19 18:36:36.157774 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x612ce00 con 0x53bba20 -5904> 2014-05-19 18:36:36.157789 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5903> 2014-05-19 18:36:36.157796 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x5fc3880 con 0x51cac60 -5902> 2014-05-19 18:36:36.157813 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5901> 2014-05-19 18:36:36.157818 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x784c380 con 0x5bd66e0 -5900> 2014-05-19 18:36:36.157832 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5899> 2014-05-19 18:36:36.157837 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 71 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x5f308c0 con 0x5bd6840 -5898> 2014-05-19 18:36:36.157851 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5897> 2014-05-19 18:36:36.157856 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.156896) v2 ==== 47+0+0 (3545138937 0 0) 0x64d3180 con 0x30594a0 -5896> 2014-05-19 18:36:36.157871 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5895> 2014-05-19 18:36:36.187157 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:41.273491) v2 ==== 47+0+0 (3290681054 0 0) 0x70eb500 con 0x5f5b760 -5894> 2014-05-19 18:36:36.187155 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:41.273491) v2 ==== 47+0+0 (3290681054 0 0) 0x70cd500 con 0x5b21600 -5893> 2014-05-19 18:36:36.187202 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.273491) v2 -- ?+0 0x71cb180 con 0x5f5b760 -5892> 2014-05-19 18:36:36.187232 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5891> 2014-05-19 18:36:36.187255 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.273491) v2 -- ?+0 0x662bc00 con 0x5b21600 -5890> 2014-05-19 18:36:36.187304 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5889> 2014-05-19 18:36:36.222548 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:39.279470) v2 ==== 47+0+0 (856868443 0 0) 0x5cbca80 con 0x618fe40 -5888> 2014-05-19 18:36:36.222594 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:39.279470) v2 -- ?+0 0x70eb500 con 0x618fe40 -5887> 2014-05-19 18:36:36.222625 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5886> 2014-05-19 18:36:36.222670 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:39.279470) v2 ==== 47+0+0 (856868443 0 0) 0x8388a80 con 0x61ae420 -5885> 2014-05-19 18:36:36.222710 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:39.279470) v2 -- ?+0 0x70cd500 con 0x61ae420 -5884> 2014-05-19 18:36:36.222743 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5883> 2014-05-19 18:36:36.371911 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:36.371711) v2 ==== 47+0+0 (3229966268 0 0) 0x5519500 con 0x4cb82c0 -5882> 2014-05-19 18:36:36.371962 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.371711) v2 -- ?+0 0x5cbca80 con 0x4cb82c0 -5881> 2014-05-19 18:36:36.371959 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:36.371711) v2 ==== 47+0+0 (3229966268 0 0) 0x788a700 con 0x4cb8160 -5880> 2014-05-19 18:36:36.372009 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5879> 2014-05-19 18:36:36.372072 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.371711) v2 -- ?+0 0x8388a80 con 0x4cb8160 -5878> 2014-05-19 18:36:36.372171 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5877> 2014-05-19 18:36:36.491358 7fd04f907700 5 osd.0 21 tick -5876> 2014-05-19 18:36:36.491379 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5875> 2014-05-19 18:36:36.491384 7fd04f907700 10 osd.0 21 do_waiters -- start -5874> 2014-05-19 18:36:36.491386 7fd04f907700 10 osd.0 21 do_waiters -- finish -5873> 2014-05-19 18:36:36.491416 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5872> 2014-05-19 18:36:36.586184 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:36.586022) v2 ==== 47+0+0 (3808075047 0 0) 0x5e6c380 con 0x4cb86e0 -5871> 2014-05-19 18:36:36.586228 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.586022) v2 -- ?+0 0x788a700 con 0x4cb86e0 -5870> 2014-05-19 18:36:36.586256 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5869> 2014-05-19 18:36:36.586246 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:36.586022) v2 ==== 47+0+0 (3808075047 0 0) 0x64a7dc0 con 0x4cb8420 -5868> 2014-05-19 18:36:36.586279 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:36.586022) v2 -- ?+0 0x5519500 con 0x4cb8420 -5867> 2014-05-19 18:36:36.586310 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5866> 2014-05-19 18:36:36.611494 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:41.697819) v2 ==== 47+0+0 (2840572636 0 0) 0x691bdc0 con 0x3094160 -5865> 2014-05-19 18:36:36.611497 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 64 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:41.697819) v2 ==== 47+0+0 (2840572636 0 0) 0x5f0aa80 con 0x30946e0 -5864> 2014-05-19 18:36:36.611541 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.697819) v2 -- ?+0 0x64a7dc0 con 0x3094160 -5863> 2014-05-19 18:36:36.611575 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5862> 2014-05-19 18:36:36.611601 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.697819) v2 -- ?+0 0x5e6c380 con 0x30946e0 -5861> 2014-05-19 18:36:36.611633 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5860> 2014-05-19 18:36:36.687676 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:41.773990) v2 ==== 47+0+0 (768637375 0 0) 0x75b0e00 con 0x5f5b760 -5859> 2014-05-19 18:36:36.687666 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:41.773990) v2 ==== 47+0+0 (768637375 0 0) 0x71236c0 con 0x5b21600 -5858> 2014-05-19 18:36:36.687722 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.773990) v2 -- ?+0 0x691bdc0 con 0x5f5b760 -5857> 2014-05-19 18:36:36.687753 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5856> 2014-05-19 18:36:36.687791 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.773990) v2 -- ?+0 0x5f0aa80 con 0x5b21600 -5855> 2014-05-19 18:36:36.687847 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5854> 2014-05-19 18:36:36.774537 7fd04e905700 20 KeyValueStore::op_tp worker waiting -5853> 2014-05-19 18:36:36.774540 7fd043ebd700 20 OSD::op_tp worker waiting -5852> 2014-05-19 18:36:36.946895 7fd0396a8700 20 OSD::command_tp worker waiting -5851> 2014-05-19 18:36:37.111931 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:42.198321) v2 ==== 47+0+0 (2214440661 0 0) 0x5e28fc0 con 0x30946e0 -5850> 2014-05-19 18:36:37.111955 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 65 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:42.198321) v2 ==== 47+0+0 (2214440661 0 0) 0x79e6a80 con 0x3094160 -5849> 2014-05-19 18:36:37.111975 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:42.198321) v2 -- ?+0 0x71236c0 con 0x30946e0 -5848> 2014-05-19 18:36:37.112013 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5847> 2014-05-19 18:36:37.112034 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:42.198321) v2 -- ?+0 0x75b0e00 con 0x3094160 -5846> 2014-05-19 18:36:37.112065 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5845> 2014-05-19 18:36:37.341029 7fd041eb9700 20 OSD::op_tp worker waiting -5844> 2014-05-19 18:36:37.341051 7fd03e6b2700 20 OSD::op_tp worker waiting -5843> 2014-05-19 18:36:37.341074 7fd03c6ae700 20 OSD::op_tp worker waiting -5842> 2014-05-19 18:36:37.341422 7fd03d6b0700 20 OSD::op_tp worker waiting -5841> 2014-05-19 18:36:37.341546 7fd0436bc700 20 OSD::op_tp worker waiting -5840> 2014-05-19 18:36:37.341660 7fd03bead700 20 OSD::op_tp worker waiting -5839> 2014-05-19 18:36:37.341840 7fd03f6b4700 20 OSD::op_tp worker waiting -5838> 2014-05-19 18:36:37.355781 7fd0446be700 20 OSD::op_tp worker waiting -5837> 2014-05-19 18:36:37.355803 7fd03feb5700 20 OSD::op_tp worker waiting -5836> 2014-05-19 18:36:37.362780 7fd040eb7700 20 OSD::op_tp worker waiting -5835> 2014-05-19 18:36:37.368110 7fd0406b6700 20 OSD::op_tp worker waiting -5834> 2014-05-19 18:36:37.389603 7fd039ea9700 20 OSD::disk_tp worker waiting -5833> 2014-05-19 18:36:37.393191 7fd03aeab700 20 OSD::op_tp worker waiting -5832> 2014-05-19 18:36:37.393282 7fd03eeb3700 20 OSD::op_tp worker waiting -5831> 2014-05-19 18:36:37.411209 7fd042ebb700 20 OSD::op_tp worker waiting -5830> 2014-05-19 18:36:37.411268 7fd03deb1700 20 OSD::op_tp worker waiting -5829> 2014-05-19 18:36:37.419766 7fd04e104700 20 KeyValueStore::op_tp worker waiting -5828> 2014-05-19 18:36:37.419769 7fd03b6ac700 20 OSD::op_tp worker waiting -5827> 2014-05-19 18:36:37.419807 7fd0426ba700 20 OSD::op_tp worker waiting -5826> 2014-05-19 18:36:37.419826 7fd0416b8700 20 OSD::op_tp worker waiting -5825> 2014-05-19 18:36:37.419852 7fd03ceaf700 20 OSD::op_tp worker waiting -5824> 2014-05-19 18:36:37.491471 7fd04f907700 5 osd.0 21 tick -5823> 2014-05-19 18:36:37.491488 7fd04f907700 7 osd.0 21 do_mon_report -5822> 2014-05-19 18:36:37.491492 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -5821> 2014-05-19 18:36:37.491496 7fd04f907700 20 osd.0 21 send_pg_stats -5820> 2014-05-19 18:36:37.491523 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5819> 2014-05-19 18:36:37.491530 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.06 < max 0.5 = yes -5818> 2014-05-19 18:36:37.491543 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -5817> 2014-05-19 18:36:37.491546 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 240.216 < min (86400 seconds) -5816> 2014-05-19 18:36:37.491564 7fd04f907700 20 osd.0 21 sched_scrub done -5815> 2014-05-19 18:36:37.491567 7fd04f907700 10 osd.0 21 do_waiters -- start -5814> 2014-05-19 18:36:37.491569 7fd04f907700 10 osd.0 21 do_waiters -- finish -5813> 2014-05-19 18:36:37.612370 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:42.698829) v2 ==== 47+0+0 (3520144671 0 0) 0x59bc540 con 0x30946e0 -5812> 2014-05-19 18:36:37.612410 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:42.698829) v2 -- ?+0 0x5e28fc0 con 0x30946e0 -5811> 2014-05-19 18:36:37.612475 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 66 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:42.698829) v2 ==== 47+0+0 (3520144671 0 0) 0x60f68c0 con 0x3094160 -5810> 2014-05-19 18:36:37.612499 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5809> 2014-05-19 18:36:37.612539 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:42.698829) v2 -- ?+0 0x79e6a80 con 0x3094160 -5808> 2014-05-19 18:36:37.612570 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5807> 2014-05-19 18:36:37.660516 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:40.717469) v2 ==== 47+0+0 (304470245 0 0) 0x6aa5500 con 0x4cb8dc0 -5806> 2014-05-19 18:36:37.660520 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:40.717469) v2 ==== 47+0+0 (304470245 0 0) 0x2ee16c0 con 0x4cb9340 -5805> 2014-05-19 18:36:37.660558 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.717469) v2 -- ?+0 0x59bc540 con 0x4cb8dc0 -5804> 2014-05-19 18:36:37.660621 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5803> 2014-05-19 18:36:37.660642 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.717469) v2 -- ?+0 0x60f68c0 con 0x4cb9340 -5802> 2014-05-19 18:36:37.660687 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5801> 2014-05-19 18:36:37.923147 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:40.980100) v2 ==== 47+0+0 (3916748962 0 0) 0x67101c0 con 0x61ae420 -5800> 2014-05-19 18:36:37.923176 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:40.980100) v2 ==== 47+0+0 (3916748962 0 0) 0x6113500 con 0x618fe40 -5799> 2014-05-19 18:36:37.923196 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.980100) v2 -- ?+0 0x6aa5500 con 0x61ae420 -5798> 2014-05-19 18:36:37.923230 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5797> 2014-05-19 18:36:37.923268 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.980100) v2 -- ?+0 0x2ee16c0 con 0x618fe40 -5796> 2014-05-19 18:36:37.923314 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5795> 2014-05-19 18:36:38.072444 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:38.072249) v2 ==== 47+0+0 (2499650554 0 0) 0x788a000 con 0x4cb8160 -5794> 2014-05-19 18:36:38.072510 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:38.072249) v2 -- ?+0 0x67101c0 con 0x4cb8160 -5793> 2014-05-19 18:36:38.072548 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5792> 2014-05-19 18:36:38.072452 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:38.072249) v2 ==== 47+0+0 (2499650554 0 0) 0x55188c0 con 0x4cb82c0 -5791> 2014-05-19 18:36:38.072623 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:38.072249) v2 -- ?+0 0x6113500 con 0x4cb82c0 -5790> 2014-05-19 18:36:38.072673 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5789> 2014-05-19 18:36:38.286706 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:38.286538) v2 ==== 47+0+0 (2453873167 0 0) 0x860fa40 con 0x4cb86e0 -5788> 2014-05-19 18:36:38.286740 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:38.286538) v2 ==== 47+0+0 (2453873167 0 0) 0x64a6700 con 0x4cb8420 -5787> 2014-05-19 18:36:38.286754 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:38.286538) v2 -- ?+0 0x788a000 con 0x4cb86e0 -5786> 2014-05-19 18:36:38.286822 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5785> 2014-05-19 18:36:38.286847 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:38.286538) v2 -- ?+0 0x55188c0 con 0x4cb8420 -5784> 2014-05-19 18:36:38.286885 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5783> 2014-05-19 18:36:38.491651 7fd04f907700 5 osd.0 21 tick -5782> 2014-05-19 18:36:38.491672 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5781> 2014-05-19 18:36:38.491676 7fd04f907700 10 osd.0 21 do_waiters -- start -5780> 2014-05-19 18:36:38.491679 7fd04f907700 10 osd.0 21 do_waiters -- finish -5779> 2014-05-19 18:36:38.491708 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5778> 2014-05-19 18:36:38.774644 7fd04e905700 20 KeyValueStore::op_tp worker waiting -5777> 2014-05-19 18:36:38.774648 7fd043ebd700 20 OSD::op_tp worker waiting -5776> 2014-05-19 18:36:38.946962 7fd0396a8700 20 OSD::command_tp worker waiting -5775> 2014-05-19 18:36:39.341104 7fd03e6b2700 20 OSD::op_tp worker waiting -5774> 2014-05-19 18:36:39.341130 7fd041eb9700 20 OSD::op_tp worker waiting -5773> 2014-05-19 18:36:39.341147 7fd03c6ae700 20 OSD::op_tp worker waiting -5772> 2014-05-19 18:36:39.341487 7fd03d6b0700 20 OSD::op_tp worker waiting -5771> 2014-05-19 18:36:39.341625 7fd0436bc700 20 OSD::op_tp worker waiting -5770> 2014-05-19 18:36:39.341727 7fd03bead700 20 OSD::op_tp worker waiting -5769> 2014-05-19 18:36:39.341920 7fd03f6b4700 20 OSD::op_tp worker waiting -5768> 2014-05-19 18:36:39.355881 7fd0446be700 20 OSD::op_tp worker waiting -5767> 2014-05-19 18:36:39.355903 7fd03feb5700 20 OSD::op_tp worker waiting -5766> 2014-05-19 18:36:39.361073 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:42.418033) v2 ==== 47+0+0 (1064236260 0 0) 0x5b28540 con 0x4cb9340 -5765> 2014-05-19 18:36:39.361123 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:42.418033) v2 -- ?+0 0x64a6700 con 0x4cb9340 -5764> 2014-05-19 18:36:39.361038 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:42.418033) v2 ==== 47+0+0 (1064236260 0 0) 0x60f6e00 con 0x4cb8dc0 -5763> 2014-05-19 18:36:39.361198 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5762> 2014-05-19 18:36:39.361253 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:42.418033) v2 -- ?+0 0x860fa40 con 0x4cb8dc0 -5761> 2014-05-19 18:36:39.361308 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5760> 2014-05-19 18:36:39.362867 7fd040eb7700 20 OSD::op_tp worker waiting -5759> 2014-05-19 18:36:39.368207 7fd0406b6700 20 OSD::op_tp worker waiting -5758> 2014-05-19 18:36:39.389698 7fd039ea9700 20 OSD::disk_tp worker waiting -5757> 2014-05-19 18:36:39.393277 7fd03aeab700 20 OSD::op_tp worker waiting -5756> 2014-05-19 18:36:39.393366 7fd03eeb3700 20 OSD::op_tp worker waiting -5755> 2014-05-19 18:36:39.411307 7fd042ebb700 20 OSD::op_tp worker waiting -5754> 2014-05-19 18:36:39.411348 7fd03deb1700 20 OSD::op_tp worker waiting -5753> 2014-05-19 18:36:39.419874 7fd04e104700 20 KeyValueStore::op_tp worker waiting -5752> 2014-05-19 18:36:39.419877 7fd03b6ac700 20 OSD::op_tp worker waiting -5751> 2014-05-19 18:36:39.419912 7fd0426ba700 20 OSD::op_tp worker waiting -5750> 2014-05-19 18:36:39.419931 7fd0416b8700 20 OSD::op_tp worker waiting -5749> 2014-05-19 18:36:39.419955 7fd03ceaf700 20 OSD::op_tp worker waiting -5748> 2014-05-19 18:36:39.491763 7fd04f907700 5 osd.0 21 tick -5747> 2014-05-19 18:36:39.491815 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5746> 2014-05-19 18:36:39.491812 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.06 < max 0.5 = yes -5745> 2014-05-19 18:36:39.491825 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -5744> 2014-05-19 18:36:39.491828 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 242.216 < min (86400 seconds) -5743> 2014-05-19 18:36:39.491847 7fd04f907700 20 osd.0 21 sched_scrub done -5742> 2014-05-19 18:36:39.491850 7fd04f907700 10 osd.0 21 do_waiters -- start -5741> 2014-05-19 18:36:39.491852 7fd04f907700 10 osd.0 21 do_waiters -- finish -5740> 2014-05-19 18:36:39.591684 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:42.648666) v2 ==== 47+0+0 (3728002103 0 0) 0x55c2380 con 0x61ae160 -5739> 2014-05-19 18:36:39.591730 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:42.648666) v2 -- ?+0 0x60f6e00 con 0x61ae160 -5738> 2014-05-19 18:36:39.591766 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5737> 2014-05-19 18:36:39.591748 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:42.648666) v2 ==== 47+0+0 (3728002103 0 0) 0x6543c00 con 0x61b0420 -5736> 2014-05-19 18:36:39.591789 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:42.648666) v2 -- ?+0 0x5b28540 con 0x61b0420 -5735> 2014-05-19 18:36:39.591821 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5734> 2014-05-19 18:36:39.907242 7fd0456c0700 10 monclient: tick -5733> 2014-05-19 18:36:39.907255 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:36:09.907254) -5732> 2014-05-19 18:36:39.907272 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:36:39.907272; renew after: 2014-05-19 18:38:19.906664) -- no -5731> 2014-05-19 18:36:39.987308 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:39.987097) v2 ==== 47+0+0 (983077866 0 0) 0x64a7a40 con 0x4cb8420 -5730> 2014-05-19 18:36:39.987313 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:39.987097) v2 ==== 47+0+0 (983077866 0 0) 0x860e000 con 0x4cb86e0 -5729> 2014-05-19 18:36:39.987362 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:39.987097) v2 -- ?+0 0x6543c00 con 0x4cb8420 -5728> 2014-05-19 18:36:39.987386 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5727> 2014-05-19 18:36:39.987406 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:39.987097) v2 -- ?+0 0x55c2380 con 0x4cb86e0 -5726> 2014-05-19 18:36:39.987556 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5725> 2014-05-19 18:36:40.257425 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -5724> 2014-05-19 18:36:40.257443 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -5723> 2014-05-19 18:36:40.257452 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x7980380 con 0x3622580 -5722> 2014-05-19 18:36:40.257479 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x6c4d6c0 con 0x3622840 -5721> 2014-05-19 18:36:40.257516 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x7970540 con 0x3622b00 -5720> 2014-05-19 18:36:40.257531 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x8388700 con 0x3622dc0 -5719> 2014-05-19 18:36:40.257545 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x6fe61c0 con 0x30594a0 -5718> 2014-05-19 18:36:40.257572 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x7a5e000 con 0x3622f20 -5717> 2014-05-19 18:36:40.257598 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x60a1a40 con 0x5e7ef20 -5716> 2014-05-19 18:36:40.257624 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x73c21c0 con 0x5e7f1e0 -5715> 2014-05-19 18:36:40.257649 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x66f8380 con 0x5e7e2c0 -5714> 2014-05-19 18:36:40.257675 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x5e288c0 con 0x5d7bb80 -5713> 2014-05-19 18:36:40.257702 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x7a5e8c0 con 0x5e7ec60 -5712> 2014-05-19 18:36:40.257729 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x70ea700 con 0x531b760 -5711> 2014-05-19 18:36:40.257754 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x784d6c0 con 0x53bba20 -5710> 2014-05-19 18:36:40.257779 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x55c8700 con 0x51cac60 -5709> 2014-05-19 18:36:40.257804 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x772cc40 con 0x5bd66e0 -5708> 2014-05-19 18:36:40.257795 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x66f8e00 con 0x3622580 -5707> 2014-05-19 18:36:40.257843 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:40.257450) v2 -- ?+0 0x6008e00 con 0x5bd6840 -5706> 2014-05-19 18:36:40.257913 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5705> 2014-05-19 18:36:40.257932 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x7122a80 con 0x3622dc0 -5704> 2014-05-19 18:36:40.257964 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5703> 2014-05-19 18:36:40.258023 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x73c2380 con 0x3622840 -5702> 2014-05-19 18:36:40.258062 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5701> 2014-05-19 18:36:40.258139 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x5e28e00 con 0x3622b00 -5700> 2014-05-19 18:36:40.258191 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5699> 2014-05-19 18:36:40.258203 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x798f880 con 0x5e7ef20 -5698> 2014-05-19 18:36:40.258221 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5697> 2014-05-19 18:36:40.258228 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x7998c40 con 0x5e7f1e0 -5696> 2014-05-19 18:36:40.258252 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5695> 2014-05-19 18:36:40.258266 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x6d60700 con 0x5e7e2c0 -5694> 2014-05-19 18:36:40.258286 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5693> 2014-05-19 18:36:40.258291 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x3511a40 con 0x5d7bb80 -5692> 2014-05-19 18:36:40.258306 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5691> 2014-05-19 18:36:40.258311 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x6b71340 con 0x5e7ec60 -5690> 2014-05-19 18:36:40.258325 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5689> 2014-05-19 18:36:40.258330 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x6009180 con 0x531b760 -5688> 2014-05-19 18:36:40.258343 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5687> 2014-05-19 18:36:40.258351 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x5cbcfc0 con 0x53bba20 -5686> 2014-05-19 18:36:40.258367 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5685> 2014-05-19 18:36:40.258376 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x662ba40 con 0x51cac60 -5684> 2014-05-19 18:36:40.258390 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5683> 2014-05-19 18:36:40.258396 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x784d880 con 0x5bd66e0 -5682> 2014-05-19 18:36:40.258412 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5681> 2014-05-19 18:36:40.258418 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 72 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x5f30e00 con 0x5bd6840 -5680> 2014-05-19 18:36:40.258432 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5679> 2014-05-19 18:36:40.258437 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x64d2a80 con 0x30594a0 -5678> 2014-05-19 18:36:40.258451 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5677> 2014-05-19 18:36:40.258456 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:40.257450) v2 ==== 47+0+0 (2897666830 0 0) 0x72e5dc0 con 0x3622f20 -5676> 2014-05-19 18:36:40.258470 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5675> 2014-05-19 18:36:40.472878 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:45.559357) v2 ==== 47+0+0 (2658322140 0 0) 0x60a0c40 con 0x31e1600 -5674> 2014-05-19 18:36:40.472881 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:45.559357) v2 ==== 47+0+0 (2658322140 0 0) 0x7a36700 con 0x61b0160 -5673> 2014-05-19 18:36:40.472933 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:45.559357) v2 -- ?+0 0x860e000 con 0x31e1600 -5672> 2014-05-19 18:36:40.472968 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5671> 2014-05-19 18:36:40.473037 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:45.559357) v2 -- ?+0 0x64a7a40 con 0x61b0160 -5670> 2014-05-19 18:36:40.473068 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5669> 2014-05-19 18:36:40.491923 7fd04f907700 5 osd.0 21 tick -5668> 2014-05-19 18:36:40.491944 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5667> 2014-05-19 18:36:40.491949 7fd04f907700 10 osd.0 21 do_waiters -- start -5666> 2014-05-19 18:36:40.491951 7fd04f907700 10 osd.0 21 do_waiters -- finish -5665> 2014-05-19 18:36:40.491975 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5664> 2014-05-19 18:36:40.774738 7fd04e905700 20 KeyValueStore::op_tp worker waiting -5663> 2014-05-19 18:36:40.774741 7fd043ebd700 20 OSD::op_tp worker waiting -5662> 2014-05-19 18:36:40.787993 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:45.874532) v2 ==== 47+0+0 (290002059 0 0) 0x55c3880 con 0x5b21600 -5661> 2014-05-19 18:36:40.788057 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:45.874532) v2 -- ?+0 0x60a0c40 con 0x5b21600 -5660> 2014-05-19 18:36:40.788093 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5659> 2014-05-19 18:36:40.788003 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:45.874532) v2 ==== 47+0+0 (290002059 0 0) 0x65436c0 con 0x5f5b760 -5658> 2014-05-19 18:36:40.788129 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:45.874532) v2 -- ?+0 0x7a36700 con 0x5f5b760 -5657> 2014-05-19 18:36:40.788161 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5656> 2014-05-19 18:36:40.947039 7fd0396a8700 20 OSD::command_tp worker waiting -5655> 2014-05-19 18:36:41.061577 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:44.118568) v2 ==== 47+0+0 (1759241164 0 0) 0x60f6700 con 0x4cb8dc0 -5654> 2014-05-19 18:36:41.061634 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:44.118568) v2 -- ?+0 0x55c3880 con 0x4cb8dc0 -5653> 2014-05-19 18:36:41.061668 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5652> 2014-05-19 18:36:41.061589 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:44.118568) v2 ==== 47+0+0 (1759241164 0 0) 0x55f5c00 con 0x4cb9340 -5651> 2014-05-19 18:36:41.061742 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:44.118568) v2 -- ?+0 0x65436c0 con 0x4cb9340 -5650> 2014-05-19 18:36:41.061784 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5649> 2014-05-19 18:36:41.292176 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:44.349182) v2 ==== 47+0+0 (2928666695 0 0) 0x55c36c0 con 0x61ae160 -5648> 2014-05-19 18:36:41.292210 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:44.349182) v2 ==== 47+0+0 (2928666695 0 0) 0x6543a40 con 0x61b0420 -5647> 2014-05-19 18:36:41.292227 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:44.349182) v2 -- ?+0 0x60f6700 con 0x61ae160 -5646> 2014-05-19 18:36:41.292259 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5645> 2014-05-19 18:36:41.292326 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:44.349182) v2 -- ?+0 0x55f5c00 con 0x61b0420 -5644> 2014-05-19 18:36:41.292369 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5643> 2014-05-19 18:36:41.341203 7fd03e6b2700 20 OSD::op_tp worker waiting -5642> 2014-05-19 18:36:41.341235 7fd03c6ae700 20 OSD::op_tp worker waiting -5641> 2014-05-19 18:36:41.341256 7fd041eb9700 20 OSD::op_tp worker waiting -5640> 2014-05-19 18:36:41.341552 7fd03d6b0700 20 OSD::op_tp worker waiting -5639> 2014-05-19 18:36:41.341698 7fd0436bc700 20 OSD::op_tp worker waiting -5638> 2014-05-19 18:36:41.341800 7fd03bead700 20 OSD::op_tp worker waiting -5637> 2014-05-19 18:36:41.341990 7fd03f6b4700 20 OSD::op_tp worker waiting -5636> 2014-05-19 18:36:41.355964 7fd0446be700 20 OSD::op_tp worker waiting -5635> 2014-05-19 18:36:41.355986 7fd03feb5700 20 OSD::op_tp worker waiting -5634> 2014-05-19 18:36:41.358017 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -5633> 2014-05-19 18:36:41.358035 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -5632> 2014-05-19 18:36:41.358043 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x6e2ba40 con 0x3622580 -5631> 2014-05-19 18:36:41.358068 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x3510380 con 0x3622840 -5630> 2014-05-19 18:36:41.358105 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x64afc00 con 0x3622b00 -5629> 2014-05-19 18:36:41.358119 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x7747180 con 0x3622dc0 -5628> 2014-05-19 18:36:41.358134 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x7931a40 con 0x30594a0 -5627> 2014-05-19 18:36:41.358177 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x5f328c0 con 0x3622f20 -5626> 2014-05-19 18:36:41.358205 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x60a0540 con 0x5e7ef20 -5625> 2014-05-19 18:36:41.358230 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x59bcc40 con 0x5e7f1e0 -5624> 2014-05-19 18:36:41.358255 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x73c28c0 con 0x5e7e2c0 -5623> 2014-05-19 18:36:41.358277 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x66f9500 con 0x5d7bb80 -5622> 2014-05-19 18:36:41.358300 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x70ea380 con 0x5e7ec60 -5621> 2014-05-19 18:36:41.358320 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x649fa40 con 0x531b760 -5620> 2014-05-19 18:36:41.358343 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x55c9a40 con 0x53bba20 -5619> 2014-05-19 18:36:41.358368 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x772ce00 con 0x51cac60 -5618> 2014-05-19 18:36:41.358392 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x6c4c000 con 0x5bd66e0 -5617> 2014-05-19 18:36:41.358416 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:41.358041) v2 -- ?+0 0x6e2a700 con 0x5bd6840 -5616> 2014-05-19 18:36:41.358407 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x78b8c40 con 0x3622580 -5615> 2014-05-19 18:36:41.358465 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5614> 2014-05-19 18:36:41.358484 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x70a9dc0 con 0x3622dc0 -5613> 2014-05-19 18:36:41.358519 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5612> 2014-05-19 18:36:41.358534 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x6e19c00 con 0x3622840 -5611> 2014-05-19 18:36:41.358552 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5610> 2014-05-19 18:36:41.358557 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x77336c0 con 0x3622b00 -5609> 2014-05-19 18:36:41.358572 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5608> 2014-05-19 18:36:41.358662 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x72e4540 con 0x3622f20 -5607> 2014-05-19 18:36:41.358718 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5606> 2014-05-19 18:36:41.358740 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x798e380 con 0x5e7ef20 -5605> 2014-05-19 18:36:41.358772 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5604> 2014-05-19 18:36:41.358790 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x7998a80 con 0x5e7f1e0 -5603> 2014-05-19 18:36:41.358820 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5602> 2014-05-19 18:36:41.358851 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x6d61500 con 0x5e7e2c0 -5601> 2014-05-19 18:36:41.358878 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5600> 2014-05-19 18:36:41.358887 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x682e700 con 0x5d7bb80 -5599> 2014-05-19 18:36:41.358901 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5598> 2014-05-19 18:36:41.358909 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x6b70380 con 0x5e7ec60 -5597> 2014-05-19 18:36:41.358923 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5596> 2014-05-19 18:36:41.358934 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x60081c0 con 0x531b760 -5595> 2014-05-19 18:36:41.358951 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5594> 2014-05-19 18:36:41.358959 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x5cbc000 con 0x53bba20 -5593> 2014-05-19 18:36:41.358973 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5592> 2014-05-19 18:36:41.358984 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x66f8700 con 0x51cac60 -5591> 2014-05-19 18:36:41.359001 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5590> 2014-05-19 18:36:41.359008 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x5667340 con 0x5bd66e0 -5589> 2014-05-19 18:36:41.359022 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5588> 2014-05-19 18:36:41.359030 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 73 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x5f31c00 con 0x5bd6840 -5587> 2014-05-19 18:36:41.359044 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5586> 2014-05-19 18:36:41.359052 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:41.358041) v2 ==== 47+0+0 (959560092 0 0) 0x5f0a1c0 con 0x30594a0 -5585> 2014-05-19 18:36:41.359067 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5584> 2014-05-19 18:36:41.362948 7fd040eb7700 20 OSD::op_tp worker waiting -5583> 2014-05-19 18:36:41.368287 7fd0406b6700 20 OSD::op_tp worker waiting -5582> 2014-05-19 18:36:41.389793 7fd039ea9700 20 OSD::disk_tp worker waiting -5581> 2014-05-19 18:36:41.393372 7fd03aeab700 20 OSD::op_tp worker waiting -5580> 2014-05-19 18:36:41.393449 7fd03eeb3700 20 OSD::op_tp worker waiting -5579> 2014-05-19 18:36:41.411404 7fd042ebb700 20 OSD::op_tp worker waiting -5578> 2014-05-19 18:36:41.411426 7fd03deb1700 20 OSD::op_tp worker waiting -5577> 2014-05-19 18:36:41.419971 7fd04e104700 20 KeyValueStore::op_tp worker waiting -5576> 2014-05-19 18:36:41.419973 7fd03b6ac700 20 OSD::op_tp worker waiting -5575> 2014-05-19 18:36:41.420012 7fd0426ba700 20 OSD::op_tp worker waiting -5574> 2014-05-19 18:36:41.420031 7fd0416b8700 20 OSD::op_tp worker waiting -5573> 2014-05-19 18:36:41.420060 7fd03ceaf700 20 OSD::op_tp worker waiting -5572> 2014-05-19 18:36:41.492023 7fd04f907700 5 osd.0 21 tick -5571> 2014-05-19 18:36:41.492043 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5570> 2014-05-19 18:36:41.492048 7fd04f907700 10 osd.0 21 do_waiters -- start -5569> 2014-05-19 18:36:41.492050 7fd04f907700 10 osd.0 21 do_waiters -- finish -5568> 2014-05-19 18:36:41.492092 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5567> 2014-05-19 18:36:41.712821 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:46.799349) v2 ==== 47+0+0 (447252865 0 0) 0x64ae700 con 0x30946e0 -5566> 2014-05-19 18:36:41.712834 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 67 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:46.799349) v2 ==== 47+0+0 (447252865 0 0) 0x63d8c40 con 0x3094160 -5565> 2014-05-19 18:36:41.712859 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:46.799349) v2 -- ?+0 0x55c36c0 con 0x30946e0 -5564> 2014-05-19 18:36:41.712891 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5563> 2014-05-19 18:36:41.712940 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:46.799349) v2 -- ?+0 0x6543a40 con 0x3094160 -5562> 2014-05-19 18:36:41.712983 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5561> 2014-05-19 18:36:42.023631 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:45.080611) v2 ==== 47+0+0 (22359685 0 0) 0x662b6c0 con 0x61ae420 -5560> 2014-05-19 18:36:42.023636 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:45.080611) v2 ==== 47+0+0 (22359685 0 0) 0x7930700 con 0x618fe40 -5559> 2014-05-19 18:36:42.023696 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:45.080611) v2 -- ?+0 0x64ae700 con 0x61ae420 -5558> 2014-05-19 18:36:42.023730 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5557> 2014-05-19 18:36:42.023756 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:45.080611) v2 -- ?+0 0x63d8c40 con 0x618fe40 -5556> 2014-05-19 18:36:42.023784 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5555> 2014-05-19 18:36:42.173087 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:42.172899) v2 ==== 47+0+0 (2615046686 0 0) 0x5519dc0 con 0x4cb82c0 -5554> 2014-05-19 18:36:42.173137 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:42.172899) v2 -- ?+0 0x7930700 con 0x4cb82c0 -5553> 2014-05-19 18:36:42.173076 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:42.172899) v2 ==== 47+0+0 (2615046686 0 0) 0x788a540 con 0x4cb8160 -5552> 2014-05-19 18:36:42.173196 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5551> 2014-05-19 18:36:42.173229 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:47.259816) v2 ==== 47+0+0 (1466133364 0 0) 0x7a37880 con 0x61b0160 -5550> 2014-05-19 18:36:42.173262 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:47.259816) v2 -- ?+0 0x5519dc0 con 0x61b0160 -5549> 2014-05-19 18:36:42.173294 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5548> 2014-05-19 18:36:42.173337 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:42.172899) v2 -- ?+0 0x662b6c0 con 0x4cb8160 -5547> 2014-05-19 18:36:42.173381 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5546> 2014-05-19 18:36:42.173396 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:47.259816) v2 ==== 47+0+0 (1466133364 0 0) 0x60a16c0 con 0x31e1600 -5545> 2014-05-19 18:36:42.173438 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:47.259816) v2 -- ?+0 0x788a540 con 0x31e1600 -5544> 2014-05-19 18:36:42.173589 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5543> 2014-05-19 18:36:42.492118 7fd04f907700 5 osd.0 21 tick -5542> 2014-05-19 18:36:42.492136 7fd04f907700 7 osd.0 21 do_mon_report -5541> 2014-05-19 18:36:42.492141 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -5540> 2014-05-19 18:36:42.492147 7fd04f907700 20 osd.0 21 send_pg_stats -5539> 2014-05-19 18:36:42.492159 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5538> 2014-05-19 18:36:42.492165 7fd04f907700 10 osd.0 21 do_waiters -- start -5537> 2014-05-19 18:36:42.492168 7fd04f907700 10 osd.0 21 do_waiters -- finish -5536> 2014-05-19 18:36:42.492197 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5535> 2014-05-19 18:36:42.774827 7fd04e905700 20 KeyValueStore::op_tp worker waiting -5534> 2014-05-19 18:36:42.774830 7fd043ebd700 20 OSD::op_tp worker waiting -5533> 2014-05-19 18:36:42.947107 7fd0396a8700 20 OSD::command_tp worker waiting -5532> 2014-05-19 18:36:42.992720 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:46.049714) v2 ==== 47+0+0 (966214183 0 0) 0x653e8c0 con 0x61ae160 -5531> 2014-05-19 18:36:42.992760 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:46.049714) v2 ==== 47+0+0 (966214183 0 0) 0x6c4d180 con 0x61b0420 -5530> 2014-05-19 18:36:42.992770 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:46.049714) v2 -- ?+0 0x60a16c0 con 0x61ae160 -5529> 2014-05-19 18:36:42.992845 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5528> 2014-05-19 18:36:42.992869 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:46.049714) v2 -- ?+0 0x7a37880 con 0x61b0420 -5527> 2014-05-19 18:36:42.992915 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5526> 2014-05-19 18:36:43.058569 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -5525> 2014-05-19 18:36:43.058587 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -5524> 2014-05-19 18:36:43.058595 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x8389880 con 0x3622580 -5523> 2014-05-19 18:36:43.058621 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x77468c0 con 0x3622840 -5522> 2014-05-19 18:36:43.058658 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x725da40 con 0x3622b00 -5521> 2014-05-19 18:36:43.058673 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x73c2540 con 0x3622dc0 -5520> 2014-05-19 18:36:43.058687 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x7998c40 con 0x30594a0 -5519> 2014-05-19 18:36:43.058714 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x798f880 con 0x3622f20 -5518> 2014-05-19 18:36:43.058744 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x5e28e00 con 0x5e7ef20 -5517> 2014-05-19 18:36:43.058771 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x73c2380 con 0x5e7f1e0 -5516> 2014-05-19 18:36:43.058796 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x7122a80 con 0x5e7e2c0 -5515> 2014-05-19 18:36:43.058821 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x66f8e00 con 0x5d7bb80 -5514> 2014-05-19 18:36:43.058845 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x64d3180 con 0x5e7ec60 -5513> 2014-05-19 18:36:43.058867 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x5f308c0 con 0x531b760 -5512> 2014-05-19 18:36:43.058891 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x784c380 con 0x53bba20 -5511> 2014-05-19 18:36:43.058916 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x5fc3880 con 0x51cac60 -5510> 2014-05-19 18:36:43.058943 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x612ce00 con 0x5bd66e0 -5509> 2014-05-19 18:36:43.058967 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:43.058593) v2 -- ?+0 0x6009dc0 con 0x5bd6840 -5508> 2014-05-19 18:36:43.058971 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x78b9500 con 0x3622580 -5507> 2014-05-19 18:36:43.059029 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5506> 2014-05-19 18:36:43.059051 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x70a81c0 con 0x3622dc0 -5505> 2014-05-19 18:36:43.059073 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5504> 2014-05-19 18:36:43.059084 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x692cc40 con 0x3622840 -5503> 2014-05-19 18:36:43.059099 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5502> 2014-05-19 18:36:43.059127 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x798e540 con 0x5e7ef20 -5501> 2014-05-19 18:36:43.059158 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5500> 2014-05-19 18:36:43.059168 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x72e4700 con 0x3622f20 -5499> 2014-05-19 18:36:43.059188 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5498> 2014-05-19 18:36:43.059214 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x711ee00 con 0x5e7e2c0 -5497> 2014-05-19 18:36:43.059248 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5496> 2014-05-19 18:36:43.059267 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x5f0b880 con 0x30594a0 -5495> 2014-05-19 18:36:43.059306 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5494> 2014-05-19 18:36:43.059314 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x7732540 con 0x3622b00 -5493> 2014-05-19 18:36:43.059347 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5492> 2014-05-19 18:36:43.059364 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x5ccc1c0 con 0x5e7f1e0 -5491> 2014-05-19 18:36:43.059389 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5490> 2014-05-19 18:36:43.059401 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x682f340 con 0x5d7bb80 -5489> 2014-05-19 18:36:43.059420 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5488> 2014-05-19 18:36:43.059431 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x6b70540 con 0x5e7ec60 -5487> 2014-05-19 18:36:43.059448 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5486> 2014-05-19 18:36:43.059455 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x6008000 con 0x531b760 -5485> 2014-05-19 18:36:43.059470 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5484> 2014-05-19 18:36:43.059478 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x70a88c0 con 0x53bba20 -5483> 2014-05-19 18:36:43.059495 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5482> 2014-05-19 18:36:43.059502 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x75b1c00 con 0x51cac60 -5481> 2014-05-19 18:36:43.059516 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5480> 2014-05-19 18:36:43.059524 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x5667a40 con 0x5bd66e0 -5479> 2014-05-19 18:36:43.059538 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5478> 2014-05-19 18:36:43.059546 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 74 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.058593) v2 ==== 47+0+0 (3526883856 0 0) 0x5f31880 con 0x5bd6840 -5477> 2014-05-19 18:36:43.059561 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5476> 2014-05-19 18:36:43.124043 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:46.181132) v2 ==== 47+0+0 (94925459 0 0) 0x73c2700 con 0x61ae420 -5475> 2014-05-19 18:36:43.124089 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:46.181132) v2 -- ?+0 0x653e8c0 con 0x61ae420 -5474> 2014-05-19 18:36:43.124127 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5473> 2014-05-19 18:36:43.124149 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:46.181132) v2 ==== 47+0+0 (94925459 0 0) 0x6009a40 con 0x618fe40 -5472> 2014-05-19 18:36:43.124186 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:46.181132) v2 -- ?+0 0x6c4d180 con 0x618fe40 -5471> 2014-05-19 18:36:43.124262 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5470> 2014-05-19 18:36:43.273634 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:43.273452) v2 ==== 47+0+0 (2233286175 0 0) 0x788b180 con 0x4cb8160 -5469> 2014-05-19 18:36:43.273635 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:43.273452) v2 ==== 47+0+0 (2233286175 0 0) 0x3438a80 con 0x4cb82c0 -5468> 2014-05-19 18:36:43.273688 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.273452) v2 -- ?+0 0x73c2700 con 0x4cb8160 -5467> 2014-05-19 18:36:43.273767 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5466> 2014-05-19 18:36:43.273790 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:43.273452) v2 -- ?+0 0x6009a40 con 0x4cb82c0 -5465> 2014-05-19 18:36:43.273879 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5464> 2014-05-19 18:36:43.341303 7fd03e6b2700 20 OSD::op_tp worker waiting -5463> 2014-05-19 18:36:43.341324 7fd03c6ae700 20 OSD::op_tp worker waiting -5462> 2014-05-19 18:36:43.341348 7fd041eb9700 20 OSD::op_tp worker waiting -5461> 2014-05-19 18:36:43.341617 7fd03d6b0700 20 OSD::op_tp worker waiting -5460> 2014-05-19 18:36:43.341767 7fd0436bc700 20 OSD::op_tp worker waiting -5459> 2014-05-19 18:36:43.341871 7fd03bead700 20 OSD::op_tp worker waiting -5458> 2014-05-19 18:36:43.342068 7fd03f6b4700 20 OSD::op_tp worker waiting -5457> 2014-05-19 18:36:43.356050 7fd0446be700 20 OSD::op_tp worker waiting -5456> 2014-05-19 18:36:43.356068 7fd03feb5700 20 OSD::op_tp worker waiting -5455> 2014-05-19 18:36:43.363028 7fd040eb7700 20 OSD::op_tp worker waiting -5454> 2014-05-19 18:36:43.368386 7fd0406b6700 20 OSD::op_tp worker waiting -5453> 2014-05-19 18:36:43.389891 7fd039ea9700 20 OSD::disk_tp worker waiting -5452> 2014-05-19 18:36:43.393470 7fd03aeab700 20 OSD::op_tp worker waiting -5451> 2014-05-19 18:36:43.393534 7fd03eeb3700 20 OSD::op_tp worker waiting -5450> 2014-05-19 18:36:43.411507 7fd042ebb700 20 OSD::op_tp worker waiting -5449> 2014-05-19 18:36:43.411564 7fd03deb1700 20 OSD::op_tp worker waiting -5448> 2014-05-19 18:36:43.420064 7fd04e104700 20 KeyValueStore::op_tp worker waiting -5447> 2014-05-19 18:36:43.420068 7fd03b6ac700 20 OSD::op_tp worker waiting -5446> 2014-05-19 18:36:43.420112 7fd0426ba700 20 OSD::op_tp worker waiting -5445> 2014-05-19 18:36:43.420134 7fd0416b8700 20 OSD::op_tp worker waiting -5444> 2014-05-19 18:36:43.420160 7fd03ceaf700 20 OSD::op_tp worker waiting -5443> 2014-05-19 18:36:43.492234 7fd04f907700 5 osd.0 21 tick -5442> 2014-05-19 18:36:43.492255 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5441> 2014-05-19 18:36:43.492259 7fd04f907700 10 osd.0 21 do_waiters -- start -5440> 2014-05-19 18:36:43.492261 7fd04f907700 10 osd.0 21 do_waiters -- finish -5439> 2014-05-19 18:36:43.492285 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5438> 2014-05-19 18:36:43.873678 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:48.960270) v2 ==== 47+0+0 (287261592 0 0) 0x6378a80 con 0x31e1600 -5437> 2014-05-19 18:36:43.873678 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:48.960270) v2 ==== 47+0+0 (287261592 0 0) 0x7a37c00 con 0x61b0160 -5436> 2014-05-19 18:36:43.873731 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.960270) v2 -- ?+0 0x788b180 con 0x31e1600 -5435> 2014-05-19 18:36:43.873763 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5434> 2014-05-19 18:36:43.873808 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.960270) v2 -- ?+0 0x3438a80 con 0x61b0160 -5433> 2014-05-19 18:36:43.873845 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5432> 2014-05-19 18:36:44.087896 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:44.087695) v2 ==== 47+0+0 (801425607 0 0) 0x860f500 con 0x4cb86e0 -5431> 2014-05-19 18:36:44.087938 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:44.087695) v2 -- ?+0 0x6378a80 con 0x4cb86e0 -5430> 2014-05-19 18:36:44.087992 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5429> 2014-05-19 18:36:44.087978 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:44.087695) v2 ==== 47+0+0 (801425607 0 0) 0x64a6540 con 0x4cb8420 -5428> 2014-05-19 18:36:44.088036 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:44.087695) v2 -- ?+0 0x7a37c00 con 0x4cb8420 -5427> 2014-05-19 18:36:44.088124 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5426> 2014-05-19 18:36:44.492325 7fd04f907700 5 osd.0 21 tick -5425> 2014-05-19 18:36:44.492373 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.05 < max 0.5 = yes -5424> 2014-05-19 18:36:44.492386 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -5423> 2014-05-19 18:36:44.492382 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5422> 2014-05-19 18:36:44.492389 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 247.216 < min (86400 seconds) -5421> 2014-05-19 18:36:44.492409 7fd04f907700 20 osd.0 21 sched_scrub done -5420> 2014-05-19 18:36:44.492412 7fd04f907700 10 osd.0 21 do_waiters -- start -5419> 2014-05-19 18:36:44.492414 7fd04f907700 10 osd.0 21 do_waiters -- finish -5418> 2014-05-19 18:36:44.774914 7fd04e905700 20 KeyValueStore::op_tp worker waiting -5417> 2014-05-19 18:36:44.774919 7fd043ebd700 20 OSD::op_tp worker waiting -5416> 2014-05-19 18:36:44.824705 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:47.881731) v2 ==== 47+0+0 (1471373337 0 0) 0x6504a80 con 0x61ae420 -5415> 2014-05-19 18:36:44.824752 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:47.881731) v2 -- ?+0 0x860f500 con 0x61ae420 -5414> 2014-05-19 18:36:44.824782 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5413> 2014-05-19 18:36:44.824761 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:47.881731) v2 ==== 47+0+0 (1471373337 0 0) 0x5519340 con 0x618fe40 -5412> 2014-05-19 18:36:44.824798 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:47.881731) v2 -- ?+0 0x64a6540 con 0x618fe40 -5411> 2014-05-19 18:36:44.824830 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5410> 2014-05-19 18:36:44.888402 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:49.975062) v2 ==== 47+0+0 (4153917051 0 0) 0x662b180 con 0x5b21600 -5409> 2014-05-19 18:36:44.888405 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:49.975062) v2 ==== 47+0+0 (4153917051 0 0) 0x68aae00 con 0x5f5b760 -5408> 2014-05-19 18:36:44.888456 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:49.975062) v2 -- ?+0 0x6504a80 con 0x5b21600 -5407> 2014-05-19 18:36:44.888487 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5406> 2014-05-19 18:36:44.888512 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:49.975062) v2 -- ?+0 0x5519340 con 0x5f5b760 -5405> 2014-05-19 18:36:44.888548 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5404> 2014-05-19 18:36:44.947182 7fd0396a8700 20 OSD::command_tp worker waiting -5403> 2014-05-19 18:36:44.974147 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:44.973980) v2 ==== 47+0+0 (978708128 0 0) 0x3438700 con 0x4cb82c0 -5402> 2014-05-19 18:36:44.974193 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:44.973980) v2 -- ?+0 0x68aae00 con 0x4cb82c0 -5401> 2014-05-19 18:36:44.974171 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:44.973980) v2 ==== 47+0+0 (978708128 0 0) 0x612ca80 con 0x4cb8160 -5400> 2014-05-19 18:36:44.974228 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5399> 2014-05-19 18:36:44.974239 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:44.973980) v2 -- ?+0 0x662b180 con 0x4cb8160 -5398> 2014-05-19 18:36:44.974276 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5397> 2014-05-19 18:36:45.162155 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:48.219186) v2 ==== 47+0+0 (1738430114 0 0) 0x5c46fc0 con 0x4cb9340 -5396> 2014-05-19 18:36:45.162170 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:48.219186) v2 ==== 47+0+0 (1738430114 0 0) 0x60f6000 con 0x4cb8dc0 -5395> 2014-05-19 18:36:45.162201 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.219186) v2 -- ?+0 0x3438700 con 0x4cb9340 -5394> 2014-05-19 18:36:45.162276 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5393> 2014-05-19 18:36:45.162312 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.219186) v2 -- ?+0 0x612ca80 con 0x4cb8dc0 -5392> 2014-05-19 18:36:45.162341 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5391> 2014-05-19 18:36:45.188417 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:45.188228) v2 ==== 47+0+0 (4169605195 0 0) 0x860e540 con 0x4cb86e0 -5390> 2014-05-19 18:36:45.188428 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:45.188228) v2 ==== 47+0+0 (4169605195 0 0) 0x799a380 con 0x4cb8420 -5389> 2014-05-19 18:36:45.188458 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:45.188228) v2 -- ?+0 0x60f6000 con 0x4cb86e0 -5388> 2014-05-19 18:36:45.188489 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5387> 2014-05-19 18:36:45.188541 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:45.188228) v2 -- ?+0 0x5c46fc0 con 0x4cb8420 -5386> 2014-05-19 18:36:45.188605 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5385> 2014-05-19 18:36:45.341395 7fd03e6b2700 20 OSD::op_tp worker waiting -5384> 2014-05-19 18:36:45.341416 7fd041eb9700 20 OSD::op_tp worker waiting -5383> 2014-05-19 18:36:45.341434 7fd03c6ae700 20 OSD::op_tp worker waiting -5382> 2014-05-19 18:36:45.341681 7fd03d6b0700 20 OSD::op_tp worker waiting -5381> 2014-05-19 18:36:45.341836 7fd0436bc700 20 OSD::op_tp worker waiting -5380> 2014-05-19 18:36:45.341940 7fd03bead700 20 OSD::op_tp worker waiting -5379> 2014-05-19 18:36:45.342137 7fd03f6b4700 20 OSD::op_tp worker waiting -5378> 2014-05-19 18:36:45.356137 7fd0446be700 20 OSD::op_tp worker waiting -5377> 2014-05-19 18:36:45.356154 7fd03feb5700 20 OSD::op_tp worker waiting -5376> 2014-05-19 18:36:45.363109 7fd040eb7700 20 OSD::op_tp worker waiting -5375> 2014-05-19 18:36:45.368481 7fd0406b6700 20 OSD::op_tp worker waiting -5374> 2014-05-19 18:36:45.389986 7fd039ea9700 20 OSD::disk_tp worker waiting -5373> 2014-05-19 18:36:45.393567 7fd03aeab700 20 OSD::op_tp worker waiting -5372> 2014-05-19 18:36:45.393612 7fd03eeb3700 20 OSD::op_tp worker waiting -5371> 2014-05-19 18:36:45.411603 7fd042ebb700 20 OSD::op_tp worker waiting -5370> 2014-05-19 18:36:45.411642 7fd03deb1700 20 OSD::op_tp worker waiting -5369> 2014-05-19 18:36:45.420159 7fd04e104700 20 KeyValueStore::op_tp worker waiting -5368> 2014-05-19 18:36:45.420162 7fd03b6ac700 20 OSD::op_tp worker waiting -5367> 2014-05-19 18:36:45.420190 7fd0426ba700 20 OSD::op_tp worker waiting -5366> 2014-05-19 18:36:45.420207 7fd0416b8700 20 OSD::op_tp worker waiting -5365> 2014-05-19 18:36:45.420225 7fd03ceaf700 20 OSD::op_tp worker waiting -5364> 2014-05-19 18:36:45.492485 7fd04f907700 5 osd.0 21 tick -5363> 2014-05-19 18:36:45.492506 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5362> 2014-05-19 18:36:45.492511 7fd04f907700 10 osd.0 21 do_waiters -- start -5361> 2014-05-19 18:36:45.492513 7fd04f907700 10 osd.0 21 do_waiters -- finish -5360> 2014-05-19 18:36:45.492537 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5359> 2014-05-19 18:36:45.813155 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:50.899835) v2 ==== 47+0+0 (730036876 0 0) 0x64a7500 con 0x3094160 -5358> 2014-05-19 18:36:45.813163 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 68 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:50.899835) v2 ==== 47+0+0 (730036876 0 0) 0x798fdc0 con 0x30946e0 -5357> 2014-05-19 18:36:45.813215 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:50.899835) v2 -- ?+0 0x799a380 con 0x3094160 -5356> 2014-05-19 18:36:45.813243 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5355> 2014-05-19 18:36:45.813260 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:50.899835) v2 -- ?+0 0x860e540 con 0x30946e0 -5354> 2014-05-19 18:36:45.813310 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5353> 2014-05-19 18:36:46.262676 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:49.319698) v2 ==== 47+0+0 (933995890 0 0) 0x5496c40 con 0x4cb9340 -5352> 2014-05-19 18:36:46.262674 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:49.319698) v2 ==== 47+0+0 (933995890 0 0) 0x5bea700 con 0x4cb8dc0 -5351> 2014-05-19 18:36:46.262719 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:49.319698) v2 -- ?+0 0x64a7500 con 0x4cb9340 -5350> 2014-05-19 18:36:46.262800 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5349> 2014-05-19 18:36:46.262821 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:49.319698) v2 -- ?+0 0x798fdc0 con 0x4cb8dc0 -5348> 2014-05-19 18:36:46.262857 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5347> 2014-05-19 18:36:46.492584 7fd04f907700 5 osd.0 21 tick -5346> 2014-05-19 18:36:46.492604 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5345> 2014-05-19 18:36:46.492609 7fd04f907700 10 osd.0 21 do_waiters -- start -5344> 2014-05-19 18:36:46.492611 7fd04f907700 10 osd.0 21 do_waiters -- finish -5343> 2014-05-19 18:36:46.492641 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5342> 2014-05-19 18:36:46.775011 7fd04e905700 20 KeyValueStore::op_tp worker waiting -5341> 2014-05-19 18:36:46.775013 7fd043ebd700 20 OSD::op_tp worker waiting -5340> 2014-05-19 18:36:46.889013 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:46.888853) v2 ==== 47+0+0 (719966593 0 0) 0x860f6c0 con 0x4cb86e0 -5339> 2014-05-19 18:36:46.889056 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:46.888853) v2 -- ?+0 0x5bea700 con 0x4cb86e0 -5338> 2014-05-19 18:36:46.889052 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:46.888853) v2 ==== 47+0+0 (719966593 0 0) 0x799a1c0 con 0x4cb8420 -5337> 2014-05-19 18:36:46.889091 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5336> 2014-05-19 18:36:46.889110 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:46.888853) v2 -- ?+0 0x5496c40 con 0x4cb8420 -5335> 2014-05-19 18:36:46.889155 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5334> 2014-05-19 18:36:46.947267 7fd0396a8700 20 OSD::command_tp worker waiting -5333> 2014-05-19 18:36:47.093208 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:50.150260) v2 ==== 47+0+0 (3497922707 0 0) 0x653efc0 con 0x61ae160 -5332> 2014-05-19 18:36:47.093215 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:50.150260) v2 ==== 47+0+0 (3497922707 0 0) 0x6c4ca80 con 0x61b0420 -5331> 2014-05-19 18:36:47.093260 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:50.150260) v2 -- ?+0 0x860f6c0 con 0x61ae160 -5330> 2014-05-19 18:36:47.093291 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5329> 2014-05-19 18:36:47.093325 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:50.150260) v2 -- ?+0 0x799a1c0 con 0x61b0420 -5328> 2014-05-19 18:36:47.093368 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5327> 2014-05-19 18:36:47.341494 7fd041eb9700 20 OSD::op_tp worker waiting -5326> 2014-05-19 18:36:47.341525 7fd03c6ae700 20 OSD::op_tp worker waiting -5325> 2014-05-19 18:36:47.341547 7fd03e6b2700 20 OSD::op_tp worker waiting -5324> 2014-05-19 18:36:47.341745 7fd03d6b0700 20 OSD::op_tp worker waiting -5323> 2014-05-19 18:36:47.341903 7fd0436bc700 20 OSD::op_tp worker waiting -5322> 2014-05-19 18:36:47.342010 7fd03bead700 20 OSD::op_tp worker waiting -5321> 2014-05-19 18:36:47.342206 7fd03f6b4700 20 OSD::op_tp worker waiting -5320> 2014-05-19 18:36:47.356229 7fd0446be700 20 OSD::op_tp worker waiting -5319> 2014-05-19 18:36:47.356251 7fd03feb5700 20 OSD::op_tp worker waiting -5318> 2014-05-19 18:36:47.363190 7fd040eb7700 20 OSD::op_tp worker waiting -5317> 2014-05-19 18:36:47.368593 7fd0406b6700 20 OSD::op_tp worker waiting -5316> 2014-05-19 18:36:47.390082 7fd039ea9700 20 OSD::disk_tp worker waiting -5315> 2014-05-19 18:36:47.393664 7fd03aeab700 20 OSD::op_tp worker waiting -5314> 2014-05-19 18:36:47.393692 7fd03eeb3700 20 OSD::op_tp worker waiting -5313> 2014-05-19 18:36:47.411700 7fd042ebb700 20 OSD::op_tp worker waiting -5312> 2014-05-19 18:36:47.411719 7fd03deb1700 20 OSD::op_tp worker waiting -5311> 2014-05-19 18:36:47.420276 7fd03b6ac700 20 OSD::op_tp worker waiting -5310> 2014-05-19 18:36:47.420276 7fd04e104700 20 KeyValueStore::op_tp worker waiting -5309> 2014-05-19 18:36:47.420305 7fd03ceaf700 20 OSD::op_tp worker waiting -5308> 2014-05-19 18:36:47.420363 7fd0426ba700 20 OSD::op_tp worker waiting -5307> 2014-05-19 18:36:47.420387 7fd0416b8700 20 OSD::op_tp worker waiting -5306> 2014-05-19 18:36:47.492679 7fd04f907700 5 osd.0 21 tick -5305> 2014-05-19 18:36:47.492696 7fd04f907700 7 osd.0 21 do_mon_report -5304> 2014-05-19 18:36:47.492700 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -5303> 2014-05-19 18:36:47.492704 7fd04f907700 20 osd.0 21 send_pg_stats -5302> 2014-05-19 18:36:47.492711 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5301> 2014-05-19 18:36:47.492715 7fd04f907700 10 osd.0 21 do_waiters -- start -5300> 2014-05-19 18:36:47.492717 7fd04f907700 10 osd.0 21 do_waiters -- finish -5299> 2014-05-19 18:36:47.492755 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5298> 2014-05-19 18:36:47.963117 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:51.020207) v2 ==== 47+0+0 (579540476 0 0) 0x662a1c0 con 0x4cb9340 -5297> 2014-05-19 18:36:47.963146 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:51.020207) v2 ==== 47+0+0 (579540476 0 0) 0x5bea380 con 0x4cb8dc0 -5296> 2014-05-19 18:36:47.963168 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:51.020207) v2 -- ?+0 0x6c4ca80 con 0x4cb9340 -5295> 2014-05-19 18:36:47.963242 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5294> 2014-05-19 18:36:47.963284 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:51.020207) v2 -- ?+0 0x653efc0 con 0x4cb8dc0 -5293> 2014-05-19 18:36:47.963332 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5292> 2014-05-19 18:36:47.973995 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:53.060789) v2 ==== 47+0+0 (875971696 0 0) 0x7a37180 con 0x61b0160 -5291> 2014-05-19 18:36:47.974053 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.060789) v2 -- ?+0 0x662a1c0 con 0x61b0160 -5290> 2014-05-19 18:36:47.974083 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5289> 2014-05-19 18:36:47.974006 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:53.060789) v2 ==== 47+0+0 (875971696 0 0) 0x2ee01c0 con 0x31e1600 -5288> 2014-05-19 18:36:47.974174 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.060789) v2 -- ?+0 0x5bea380 con 0x31e1600 -5287> 2014-05-19 18:36:47.974208 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5286> 2014-05-19 18:36:48.193775 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:51.250829) v2 ==== 47+0+0 (1127037467 0 0) 0x6c4c700 con 0x61b0420 -5285> 2014-05-19 18:36:48.193808 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:51.250829) v2 ==== 47+0+0 (1127037467 0 0) 0x6efefc0 con 0x61ae160 -5284> 2014-05-19 18:36:48.193820 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:51.250829) v2 -- ?+0 0x7a37180 con 0x61b0420 -5283> 2014-05-19 18:36:48.193871 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5282> 2014-05-19 18:36:48.193894 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:51.250829) v2 -- ?+0 0x2ee01c0 con 0x61ae160 -5281> 2014-05-19 18:36:48.193929 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5280> 2014-05-19 18:36:48.492796 7fd04f907700 5 osd.0 21 tick -5279> 2014-05-19 18:36:48.492816 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5278> 2014-05-19 18:36:48.492821 7fd04f907700 10 osd.0 21 do_waiters -- start -5277> 2014-05-19 18:36:48.492823 7fd04f907700 10 osd.0 21 do_waiters -- finish -5276> 2014-05-19 18:36:48.492853 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5275> 2014-05-19 18:36:48.775097 7fd04e905700 20 KeyValueStore::op_tp worker waiting -5274> 2014-05-19 18:36:48.775106 7fd043ebd700 20 OSD::op_tp worker waiting -5273> 2014-05-19 18:36:48.947353 7fd0396a8700 20 OSD::command_tp worker waiting -5272> 2014-05-19 18:36:48.959106 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -5271> 2014-05-19 18:36:48.959125 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -5270> 2014-05-19 18:36:48.959133 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x6c4a000 con 0x3622580 -5269> 2014-05-19 18:36:48.959158 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x3511180 con 0x3622840 -5268> 2014-05-19 18:36:48.959196 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x79988c0 con 0x3622b00 -5267> 2014-05-19 18:36:48.959210 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x7a5ea80 con 0x3622dc0 -5266> 2014-05-19 18:36:48.959224 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x73c3a40 con 0x30594a0 -5265> 2014-05-19 18:36:48.959248 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x72e4fc0 con 0x3622f20 -5264> 2014-05-19 18:36:48.959278 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x6fe7880 con 0x5e7ef20 -5263> 2014-05-19 18:36:48.959306 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x60a0700 con 0x5e7f1e0 -5262> 2014-05-19 18:36:48.959332 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x5e28c40 con 0x5e7e2c0 -5261> 2014-05-19 18:36:48.959371 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x66f88c0 con 0x5d7bb80 -5260> 2014-05-19 18:36:48.959400 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x70ebdc0 con 0x5e7ec60 -5259> 2014-05-19 18:36:48.959425 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x784ca80 con 0x531b760 -5258> 2014-05-19 18:36:48.959452 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x6543500 con 0x53bba20 -5257> 2014-05-19 18:36:48.959475 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x70cc1c0 con 0x51cac60 -5256> 2014-05-19 18:36:48.959481 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x78b9a40 con 0x3622580 -5255> 2014-05-19 18:36:48.959499 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x60096c0 con 0x5bd66e0 -5254> 2014-05-19 18:36:48.959522 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:48.959131) v2 -- ?+0 0x6c4a700 con 0x5bd6840 -5253> 2014-05-19 18:36:48.959567 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5252> 2014-05-19 18:36:48.959679 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x6711500 con 0x3622b00 -5251> 2014-05-19 18:36:48.959731 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5250> 2014-05-19 18:36:48.959758 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x692d500 con 0x3622840 -5249> 2014-05-19 18:36:48.959791 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5248> 2014-05-19 18:36:48.959806 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x7970e00 con 0x5e7ef20 -5247> 2014-05-19 18:36:48.959836 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5246> 2014-05-19 18:36:48.959849 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x5ccc8c0 con 0x5e7f1e0 -5245> 2014-05-19 18:36:48.959877 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5244> 2014-05-19 18:36:48.959893 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x711fa40 con 0x5e7e2c0 -5243> 2014-05-19 18:36:48.959922 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5242> 2014-05-19 18:36:48.959935 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x51a68c0 con 0x5d7bb80 -5241> 2014-05-19 18:36:48.959965 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5240> 2014-05-19 18:36:48.959978 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x6008540 con 0x531b760 -5239> 2014-05-19 18:36:48.960028 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5238> 2014-05-19 18:36:48.960042 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x691b6c0 con 0x53bba20 -5237> 2014-05-19 18:36:48.960073 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5236> 2014-05-19 18:36:48.960086 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x503e000 con 0x51cac60 -5235> 2014-05-19 18:36:48.960116 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5234> 2014-05-19 18:36:48.960132 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x5666c40 con 0x5bd66e0 -5233> 2014-05-19 18:36:48.960162 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5232> 2014-05-19 18:36:48.960175 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 75 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x5f30540 con 0x5bd6840 -5231> 2014-05-19 18:36:48.960204 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5230> 2014-05-19 18:36:48.960221 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x76acfc0 con 0x30594a0 -5229> 2014-05-19 18:36:48.960251 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5228> 2014-05-19 18:36:48.960263 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x72e5500 con 0x3622f20 -5227> 2014-05-19 18:36:48.960293 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5226> 2014-05-19 18:36:48.960306 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x73c0000 con 0x3622dc0 -5225> 2014-05-19 18:36:48.960333 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5224> 2014-05-19 18:36:48.960348 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:48.959131) v2 ==== 47+0+0 (2926260862 0 0) 0x6b716c0 con 0x5e7ec60 -5223> 2014-05-19 18:36:48.960377 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5222> 2014-05-19 18:36:49.074422 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:54.161243) v2 ==== 47+0+0 (2150794913 0 0) 0x60e2000 con 0x31e1600 -5221> 2014-05-19 18:36:49.074483 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:54.161243) v2 -- ?+0 0x6efefc0 con 0x31e1600 -5220> 2014-05-19 18:36:49.074521 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5219> 2014-05-19 18:36:49.074434 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:54.161243) v2 ==== 47+0+0 (2150794913 0 0) 0x60a0e00 con 0x61b0160 -5218> 2014-05-19 18:36:49.074554 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:54.161243) v2 -- ?+0 0x6c4c700 con 0x61b0160 -5217> 2014-05-19 18:36:49.074585 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5216> 2014-05-19 18:36:49.341594 7fd041eb9700 20 OSD::op_tp worker waiting -5215> 2014-05-19 18:36:49.341615 7fd03c6ae700 20 OSD::op_tp worker waiting -5214> 2014-05-19 18:36:49.341634 7fd03e6b2700 20 OSD::op_tp worker waiting -5213> 2014-05-19 18:36:49.341810 7fd03d6b0700 20 OSD::op_tp worker waiting -5212> 2014-05-19 18:36:49.341973 7fd0436bc700 20 OSD::op_tp worker waiting -5211> 2014-05-19 18:36:49.342080 7fd03bead700 20 OSD::op_tp worker waiting -5210> 2014-05-19 18:36:49.342276 7fd03f6b4700 20 OSD::op_tp worker waiting -5209> 2014-05-19 18:36:49.356312 7fd0446be700 20 OSD::op_tp worker waiting -5208> 2014-05-19 18:36:49.356337 7fd03feb5700 20 OSD::op_tp worker waiting -5207> 2014-05-19 18:36:49.363279 7fd040eb7700 20 OSD::op_tp worker waiting -5206> 2014-05-19 18:36:49.368689 7fd0406b6700 20 OSD::op_tp worker waiting -5205> 2014-05-19 18:36:49.390179 7fd039ea9700 20 OSD::disk_tp worker waiting -5204> 2014-05-19 18:36:49.393772 7fd03aeab700 20 OSD::op_tp worker waiting -5203> 2014-05-19 18:36:49.393830 7fd03eeb3700 20 OSD::op_tp worker waiting -5202> 2014-05-19 18:36:49.411801 7fd042ebb700 20 OSD::op_tp worker waiting -5201> 2014-05-19 18:36:49.411859 7fd03deb1700 20 OSD::op_tp worker waiting -5200> 2014-05-19 18:36:49.420376 7fd04e104700 20 KeyValueStore::op_tp worker waiting -5199> 2014-05-19 18:36:49.420379 7fd03b6ac700 20 OSD::op_tp worker waiting -5198> 2014-05-19 18:36:49.420406 7fd03ceaf700 20 OSD::op_tp worker waiting -5197> 2014-05-19 18:36:49.420434 7fd0426ba700 20 OSD::op_tp worker waiting -5196> 2014-05-19 18:36:49.420467 7fd0416b8700 20 OSD::op_tp worker waiting -5195> 2014-05-19 18:36:49.492910 7fd04f907700 5 osd.0 21 tick -5194> 2014-05-19 18:36:49.492931 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5193> 2014-05-19 18:36:49.492936 7fd04f907700 10 osd.0 21 do_waiters -- start -5192> 2014-05-19 18:36:49.492938 7fd04f907700 10 osd.0 21 do_waiters -- finish -5191> 2014-05-19 18:36:49.492968 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5190> 2014-05-19 18:36:49.894332 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:52.951357) v2 ==== 47+0+0 (2512202559 0 0) 0x6c4da40 con 0x61b0420 -5189> 2014-05-19 18:36:49.894327 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:52.951357) v2 ==== 47+0+0 (2512202559 0 0) 0x6efe540 con 0x61ae160 -5188> 2014-05-19 18:36:49.894374 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:52.951357) v2 -- ?+0 0x60a0e00 con 0x61b0420 -5187> 2014-05-19 18:36:49.894404 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5186> 2014-05-19 18:36:49.894475 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:52.951357) v2 -- ?+0 0x60e2000 con 0x61ae160 -5185> 2014-05-19 18:36:49.894511 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5184> 2014-05-19 18:36:49.907376 7fd0456c0700 10 monclient: tick -5183> 2014-05-19 18:36:49.907391 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:36:19.907389) -5182> 2014-05-19 18:36:49.907408 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:36:49.907407; renew after: 2014-05-19 18:38:19.906664) -- no -5181> 2014-05-19 18:36:50.188746 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:55.275611) v2 ==== 47+0+0 (1492252725 0 0) 0x68aa1c0 con 0x5f5b760 -5180> 2014-05-19 18:36:50.188799 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:55.275611) v2 -- ?+0 0x6c4da40 con 0x5f5b760 -5179> 2014-05-19 18:36:50.188757 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:55.275611) v2 ==== 47+0+0 (1492252725 0 0) 0x36f7180 con 0x5b21600 -5178> 2014-05-19 18:36:50.188905 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5177> 2014-05-19 18:36:50.188942 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:55.275611) v2 -- ?+0 0x6efe540 con 0x5b21600 -5176> 2014-05-19 18:36:50.188975 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5175> 2014-05-19 18:36:50.493022 7fd04f907700 5 osd.0 21 tick -5174> 2014-05-19 18:36:50.493071 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.05 < max 0.5 = yes -5173> 2014-05-19 18:36:50.493083 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -5172> 2014-05-19 18:36:50.493079 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5171> 2014-05-19 18:36:50.493087 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 253.217 < min (86400 seconds) -5170> 2014-05-19 18:36:50.493105 7fd04f907700 20 osd.0 21 sched_scrub done -5169> 2014-05-19 18:36:50.493108 7fd04f907700 10 osd.0 21 do_waiters -- start -5168> 2014-05-19 18:36:50.493110 7fd04f907700 10 osd.0 21 do_waiters -- finish -5167> 2014-05-19 18:36:50.725077 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:53.782182) v2 ==== 47+0+0 (2788264574 0 0) 0x5deaa80 con 0x61ae420 -5166> 2014-05-19 18:36:50.725097 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:53.782182) v2 ==== 47+0+0 (2788264574 0 0) 0x5519dc0 con 0x618fe40 -5165> 2014-05-19 18:36:50.725124 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.782182) v2 -- ?+0 0x36f7180 con 0x61ae420 -5164> 2014-05-19 18:36:50.725202 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5163> 2014-05-19 18:36:50.725243 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.782182) v2 -- ?+0 0x68aa1c0 con 0x618fe40 -5162> 2014-05-19 18:36:50.725281 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5161> 2014-05-19 18:36:50.774858 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:55.861714) v2 ==== 47+0+0 (2276225450 0 0) 0x72dfdc0 con 0x31e1600 -5160> 2014-05-19 18:36:50.774866 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:55.861714) v2 ==== 47+0+0 (2276225450 0 0) 0x2f4efc0 con 0x61b0160 -5159> 2014-05-19 18:36:50.774897 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:55.861714) v2 -- ?+0 0x5deaa80 con 0x31e1600 -5158> 2014-05-19 18:36:50.774935 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5157> 2014-05-19 18:36:50.774976 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:55.861714) v2 -- ?+0 0x5519dc0 con 0x61b0160 -5156> 2014-05-19 18:36:50.775018 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5155> 2014-05-19 18:36:50.775161 7fd04e905700 20 KeyValueStore::op_tp worker waiting -5154> 2014-05-19 18:36:50.775172 7fd043ebd700 20 OSD::op_tp worker waiting -5153> 2014-05-19 18:36:50.874710 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:50.874532) v2 ==== 47+0+0 (979219018 0 0) 0x612dc00 con 0x4cb8160 -5152> 2014-05-19 18:36:50.874722 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:50.874532) v2 ==== 47+0+0 (979219018 0 0) 0x3438380 con 0x4cb82c0 -5151> 2014-05-19 18:36:50.874760 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:50.874532) v2 -- ?+0 0x72dfdc0 con 0x4cb8160 -5150> 2014-05-19 18:36:50.874863 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5149> 2014-05-19 18:36:50.874915 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:50.874532) v2 -- ?+0 0x2f4efc0 con 0x4cb82c0 -5148> 2014-05-19 18:36:50.874958 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5147> 2014-05-19 18:36:50.947433 7fd0396a8700 20 OSD::command_tp worker waiting -5146> 2014-05-19 18:36:51.113495 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:56.200378) v2 ==== 47+0+0 (2841662116 0 0) 0x5fc3180 con 0x30946e0 -5145> 2014-05-19 18:36:51.113506 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 69 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:56.200378) v2 ==== 47+0+0 (2841662116 0 0) 0x64a7dc0 con 0x3094160 -5144> 2014-05-19 18:36:51.113543 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:56.200378) v2 -- ?+0 0x612dc00 con 0x30946e0 -5143> 2014-05-19 18:36:51.113575 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5142> 2014-05-19 18:36:51.113619 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:56.200378) v2 -- ?+0 0x3438380 con 0x3094160 -5141> 2014-05-19 18:36:51.113697 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5140> 2014-05-19 18:36:51.341693 7fd041eb9700 20 OSD::op_tp worker waiting -5139> 2014-05-19 18:36:51.341713 7fd03e6b2700 20 OSD::op_tp worker waiting -5138> 2014-05-19 18:36:51.341731 7fd03c6ae700 20 OSD::op_tp worker waiting -5137> 2014-05-19 18:36:51.341874 7fd03d6b0700 20 OSD::op_tp worker waiting -5136> 2014-05-19 18:36:51.342042 7fd0436bc700 20 OSD::op_tp worker waiting -5135> 2014-05-19 18:36:51.342150 7fd03bead700 20 OSD::op_tp worker waiting -5134> 2014-05-19 18:36:51.342348 7fd03f6b4700 20 OSD::op_tp worker waiting -5133> 2014-05-19 18:36:51.356399 7fd0446be700 20 OSD::op_tp worker waiting -5132> 2014-05-19 18:36:51.356419 7fd03feb5700 20 OSD::op_tp worker waiting -5131> 2014-05-19 18:36:51.363375 7fd040eb7700 20 OSD::op_tp worker waiting -5130> 2014-05-19 18:36:51.368769 7fd0406b6700 20 OSD::op_tp worker waiting -5129> 2014-05-19 18:36:51.390275 7fd039ea9700 20 OSD::disk_tp worker waiting -5128> 2014-05-19 18:36:51.393856 7fd03aeab700 20 OSD::op_tp worker waiting -5127> 2014-05-19 18:36:51.393912 7fd03eeb3700 20 OSD::op_tp worker waiting -5126> 2014-05-19 18:36:51.411884 7fd042ebb700 20 OSD::op_tp worker waiting -5125> 2014-05-19 18:36:51.411941 7fd03deb1700 20 OSD::op_tp worker waiting -5124> 2014-05-19 18:36:51.420463 7fd04e104700 20 KeyValueStore::op_tp worker waiting -5123> 2014-05-19 18:36:51.420470 7fd03b6ac700 20 OSD::op_tp worker waiting -5122> 2014-05-19 18:36:51.420488 7fd0426ba700 20 OSD::op_tp worker waiting -5121> 2014-05-19 18:36:51.420513 7fd03ceaf700 20 OSD::op_tp worker waiting -5120> 2014-05-19 18:36:51.420544 7fd0416b8700 20 OSD::op_tp worker waiting -5119> 2014-05-19 18:36:51.493187 7fd04f907700 5 osd.0 21 tick -5118> 2014-05-19 18:36:51.493208 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5117> 2014-05-19 18:36:51.493213 7fd04f907700 10 osd.0 21 do_waiters -- start -5116> 2014-05-19 18:36:51.493215 7fd04f907700 10 osd.0 21 do_waiters -- finish -5115> 2014-05-19 18:36:51.493245 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5114> 2014-05-19 18:36:51.889189 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:56.976102) v2 ==== 47+0+0 (1830745481 0 0) 0x5c46380 con 0x5b21600 -5113> 2014-05-19 18:36:51.889192 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:56.976102) v2 ==== 47+0+0 (1830745481 0 0) 0x5beaa80 con 0x5f5b760 -5112> 2014-05-19 18:36:51.889251 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:56.976102) v2 -- ?+0 0x5fc3180 con 0x5b21600 -5111> 2014-05-19 18:36:51.889286 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5110> 2014-05-19 18:36:51.889338 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:56.976102) v2 -- ?+0 0x64a7dc0 con 0x5f5b760 -5109> 2014-05-19 18:36:51.889369 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5108> 2014-05-19 18:36:52.493276 7fd04f907700 5 osd.0 21 tick -5107> 2014-05-19 18:36:52.493293 7fd04f907700 7 osd.0 21 do_mon_report -5106> 2014-05-19 18:36:52.493297 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -5105> 2014-05-19 18:36:52.493301 7fd04f907700 20 osd.0 21 send_pg_stats -5104> 2014-05-19 18:36:52.493309 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -5103> 2014-05-19 18:36:52.493312 7fd04f907700 10 osd.0 21 do_waiters -- start -5102> 2014-05-19 18:36:52.493314 7fd04f907700 10 osd.0 21 do_waiters -- finish -5101> 2014-05-19 18:36:52.493345 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5100> 2014-05-19 18:36:52.775238 7fd04e905700 20 KeyValueStore::op_tp worker waiting -5099> 2014-05-19 18:36:52.775240 7fd043ebd700 20 OSD::op_tp worker waiting -5098> 2014-05-19 18:36:52.789610 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:52.789405) v2 ==== 47+0+0 (3000791405 0 0) 0x799bdc0 con 0x4cb8420 -5097> 2014-05-19 18:36:52.789612 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:52.789405) v2 ==== 47+0+0 (3000791405 0 0) 0x2e48e00 con 0x4cb86e0 -5096> 2014-05-19 18:36:52.789660 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:52.789405) v2 -- ?+0 0x5beaa80 con 0x4cb8420 -5095> 2014-05-19 18:36:52.789699 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5094> 2014-05-19 18:36:52.789750 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:52.789405) v2 -- ?+0 0x5c46380 con 0x4cb86e0 -5093> 2014-05-19 18:36:52.789781 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5092> 2014-05-19 18:36:52.813831 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:57.900811) v2 ==== 47+0+0 (2897545427 0 0) 0x75b0e00 con 0x30946e0 -5091> 2014-05-19 18:36:52.813859 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 70 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:57.900811) v2 ==== 47+0+0 (2897545427 0 0) 0x71236c0 con 0x3094160 -5090> 2014-05-19 18:36:52.813875 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:57.900811) v2 -- ?+0 0x2e48e00 con 0x30946e0 -5089> 2014-05-19 18:36:52.813921 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5088> 2014-05-19 18:36:52.813988 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:57.900811) v2 -- ?+0 0x799bdc0 con 0x3094160 -5087> 2014-05-19 18:36:52.814106 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5086> 2014-05-19 18:36:52.947517 7fd0396a8700 20 OSD::command_tp worker waiting -5085> 2014-05-19 18:36:52.989622 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:58.076598) v2 ==== 47+0+0 (3948706005 0 0) 0x2ee1500 con 0x5b21600 -5084> 2014-05-19 18:36:52.989644 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:58.076598) v2 ==== 47+0+0 (3948706005 0 0) 0x66f81c0 con 0x5f5b760 -5083> 2014-05-19 18:36:52.989682 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.076598) v2 -- ?+0 0x75b0e00 con 0x5b21600 -5082> 2014-05-19 18:36:52.989761 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5081> 2014-05-19 18:36:52.989872 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.076598) v2 -- ?+0 0x71236c0 con 0x5f5b760 -5080> 2014-05-19 18:36:52.989903 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5079> 2014-05-19 18:36:53.341785 7fd041eb9700 20 OSD::op_tp worker waiting -5078> 2014-05-19 18:36:53.341837 7fd03e6b2700 20 OSD::op_tp worker waiting -5077> 2014-05-19 18:36:53.341860 7fd03c6ae700 20 OSD::op_tp worker waiting -5076> 2014-05-19 18:36:53.341937 7fd03d6b0700 20 OSD::op_tp worker waiting -5075> 2014-05-19 18:36:53.342101 7fd0436bc700 20 OSD::op_tp worker waiting -5074> 2014-05-19 18:36:53.342220 7fd03bead700 20 OSD::op_tp worker waiting -5073> 2014-05-19 18:36:53.342416 7fd03f6b4700 20 OSD::op_tp worker waiting -5072> 2014-05-19 18:36:53.356487 7fd0446be700 20 OSD::op_tp worker waiting -5071> 2014-05-19 18:36:53.356508 7fd03feb5700 20 OSD::op_tp worker waiting -5070> 2014-05-19 18:36:53.363464 7fd040eb7700 20 OSD::op_tp worker waiting -5069> 2014-05-19 18:36:53.368866 7fd0406b6700 20 OSD::op_tp worker waiting -5068> 2014-05-19 18:36:53.390371 7fd039ea9700 20 OSD::disk_tp worker waiting -5067> 2014-05-19 18:36:53.393939 7fd03aeab700 20 OSD::op_tp worker waiting -5066> 2014-05-19 18:36:53.393989 7fd03eeb3700 20 OSD::op_tp worker waiting -5065> 2014-05-19 18:36:53.411976 7fd042ebb700 20 OSD::op_tp worker waiting -5064> 2014-05-19 18:36:53.412019 7fd03deb1700 20 OSD::op_tp worker waiting -5063> 2014-05-19 18:36:53.420549 7fd04e104700 20 KeyValueStore::op_tp worker waiting -5062> 2014-05-19 18:36:53.420554 7fd03b6ac700 20 OSD::op_tp worker waiting -5061> 2014-05-19 18:36:53.420572 7fd0426ba700 20 OSD::op_tp worker waiting -5060> 2014-05-19 18:36:53.420615 7fd03ceaf700 20 OSD::op_tp worker waiting -5059> 2014-05-19 18:36:53.420639 7fd0416b8700 20 OSD::op_tp worker waiting -5058> 2014-05-19 18:36:53.493398 7fd04f907700 5 osd.0 21 tick -5057> 2014-05-19 18:36:53.493446 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.04 < max 0.5 = yes -5056> 2014-05-19 18:36:53.493459 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -5055> 2014-05-19 18:36:53.493456 7fd03a6aa700 20 OSD::recovery_tp worker waiting -5054> 2014-05-19 18:36:53.493462 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 256.217 < min (86400 seconds) -5053> 2014-05-19 18:36:53.493481 7fd04f907700 20 osd.0 21 sched_scrub done -5052> 2014-05-19 18:36:53.493484 7fd04f907700 10 osd.0 21 do_waiters -- start -5051> 2014-05-19 18:36:53.493486 7fd04f907700 10 osd.0 21 do_waiters -- finish -5050> 2014-05-19 18:36:53.659660 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -5049> 2014-05-19 18:36:53.659678 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -5048> 2014-05-19 18:36:53.659686 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x77476c0 con 0x3622580 -5047> 2014-05-19 18:36:53.659713 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x72e48c0 con 0x3622840 -5046> 2014-05-19 18:36:53.659750 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x5f32380 con 0x3622b00 -5045> 2014-05-19 18:36:53.659765 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x3511500 con 0x3622dc0 -5044> 2014-05-19 18:36:53.659780 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x711ee00 con 0x30594a0 -5043> 2014-05-19 18:36:53.659806 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x72e4700 con 0x3622f20 -5042> 2014-05-19 18:36:53.659834 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x798e540 con 0x5e7ef20 -5041> 2014-05-19 18:36:53.659860 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x692cc40 con 0x5e7f1e0 -5040> 2014-05-19 18:36:53.659888 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x70a81c0 con 0x5e7e2c0 -5039> 2014-05-19 18:36:53.659910 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x78b9500 con 0x5d7bb80 -5038> 2014-05-19 18:36:53.659934 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x5f0a1c0 con 0x5e7ec60 -5037> 2014-05-19 18:36:53.659959 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x5f31c00 con 0x531b760 -5036> 2014-05-19 18:36:53.659983 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x5667340 con 0x53bba20 -5035> 2014-05-19 18:36:53.660009 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x66f8700 con 0x51cac60 -5034> 2014-05-19 18:36:53.660034 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x5cbc000 con 0x5bd66e0 -5033> 2014-05-19 18:36:53.660040 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x5dea8c0 con 0x3622580 -5032> 2014-05-19 18:36:53.660057 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:53.659685) v2 -- ?+0 0x60081c0 con 0x5bd6840 -5031> 2014-05-19 18:36:53.660096 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5030> 2014-05-19 18:36:53.660110 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x5c04e00 con 0x3622b00 -5029> 2014-05-19 18:36:53.660142 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5028> 2014-05-19 18:36:53.660164 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x74a6700 con 0x3622dc0 -5027> 2014-05-19 18:36:53.660236 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -5026> 2014-05-19 18:36:53.660271 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x692d340 con 0x3622840 -5025> 2014-05-19 18:36:53.660307 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -5024> 2014-05-19 18:36:53.660396 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x79716c0 con 0x5e7ef20 -5023> 2014-05-19 18:36:53.660443 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5022> 2014-05-19 18:36:53.660461 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x5f7b180 con 0x5e7f1e0 -5021> 2014-05-19 18:36:53.660488 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -5020> 2014-05-19 18:36:53.660508 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x711e8c0 con 0x5e7e2c0 -5019> 2014-05-19 18:36:53.660539 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5018> 2014-05-19 18:36:53.660552 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x51a6a80 con 0x5d7bb80 -5017> 2014-05-19 18:36:53.660606 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -5016> 2014-05-19 18:36:53.660622 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x6b71dc0 con 0x5e7ec60 -5015> 2014-05-19 18:36:53.660654 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5014> 2014-05-19 18:36:53.660667 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x60088c0 con 0x531b760 -5013> 2014-05-19 18:36:53.660693 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -5012> 2014-05-19 18:36:53.660711 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x5dff880 con 0x53bba20 -5011> 2014-05-19 18:36:53.660744 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5010> 2014-05-19 18:36:53.660758 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x669ee00 con 0x51cac60 -5009> 2014-05-19 18:36:53.660788 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -5008> 2014-05-19 18:36:53.660803 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x5666540 con 0x5bd66e0 -5007> 2014-05-19 18:36:53.660833 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5006> 2014-05-19 18:36:53.660845 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 76 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x7890700 con 0x5bd6840 -5005> 2014-05-19 18:36:53.660874 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -5004> 2014-05-19 18:36:53.660886 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x76adc00 con 0x30594a0 -5003> 2014-05-19 18:36:53.660916 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5002> 2014-05-19 18:36:53.660929 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:53.659685) v2 ==== 47+0+0 (1861086112 0 0) 0x72e5c00 con 0x3622f20 -5001> 2014-05-19 18:36:53.660958 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -5000> 2014-05-19 18:36:53.863595 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:56.920737) v2 ==== 47+0+0 (155379693 0 0) 0x2f4efc0 con 0x4cb9340 -4999> 2014-05-19 18:36:53.863649 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:56.920737) v2 -- ?+0 0x66f81c0 con 0x4cb9340 -4998> 2014-05-19 18:36:53.863649 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:56.920737) v2 ==== 47+0+0 (155379693 0 0) 0x55f56c0 con 0x4cb8dc0 -4997> 2014-05-19 18:36:53.863684 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4996> 2014-05-19 18:36:53.863722 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:56.920737) v2 -- ?+0 0x2ee1500 con 0x4cb8dc0 -4995> 2014-05-19 18:36:53.863778 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4994> 2014-05-19 18:36:53.914314 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:59.001338) v2 ==== 47+0+0 (3327381235 0 0) 0x75b08c0 con 0x30946e0 -4993> 2014-05-19 18:36:53.914361 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:59.001338) v2 -- ?+0 0x55f56c0 con 0x30946e0 -4992> 2014-05-19 18:36:53.914362 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 71 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:59.001338) v2 ==== 47+0+0 (3327381235 0 0) 0x5e6c380 con 0x3094160 -4991> 2014-05-19 18:36:53.914395 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4990> 2014-05-19 18:36:53.914407 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:59.001338) v2 -- ?+0 0x2f4efc0 con 0x3094160 -4989> 2014-05-19 18:36:53.914510 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4988> 2014-05-19 18:36:54.493568 7fd04f907700 5 osd.0 21 tick -4987> 2014-05-19 18:36:54.493620 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4986> 2014-05-19 18:36:54.493617 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.04 < max 0.5 = yes -4985> 2014-05-19 18:36:54.493630 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -4984> 2014-05-19 18:36:54.493633 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 257.218 < min (86400 seconds) -4983> 2014-05-19 18:36:54.493651 7fd04f907700 20 osd.0 21 sched_scrub done -4982> 2014-05-19 18:36:54.493654 7fd04f907700 10 osd.0 21 do_waiters -- start -4981> 2014-05-19 18:36:54.493656 7fd04f907700 10 osd.0 21 do_waiters -- finish -4980> 2014-05-19 18:36:54.690085 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:59.777101) v2 ==== 47+0+0 (525703303 0 0) 0x5e28380 con 0x5f5b760 -4979> 2014-05-19 18:36:54.690086 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:59.777101) v2 ==== 47+0+0 (525703303 0 0) 0x511c380 con 0x5b21600 -4978> 2014-05-19 18:36:54.690140 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:59.777101) v2 -- ?+0 0x5e6c380 con 0x5f5b760 -4977> 2014-05-19 18:36:54.690191 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4976> 2014-05-19 18:36:54.690252 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:59.777101) v2 -- ?+0 0x75b08c0 con 0x5b21600 -4975> 2014-05-19 18:36:54.690291 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4974> 2014-05-19 18:36:54.775329 7fd04e905700 20 KeyValueStore::op_tp worker waiting -4973> 2014-05-19 18:36:54.775332 7fd043ebd700 20 OSD::op_tp worker waiting -4972> 2014-05-19 18:36:54.947614 7fd0396a8700 20 OSD::command_tp worker waiting -4971> 2014-05-19 18:36:55.341881 7fd041eb9700 20 OSD::op_tp worker waiting -4970> 2014-05-19 18:36:55.341905 7fd03e6b2700 20 OSD::op_tp worker waiting -4969> 2014-05-19 18:36:55.341933 7fd03c6ae700 20 OSD::op_tp worker waiting -4968> 2014-05-19 18:36:55.342000 7fd03d6b0700 20 OSD::op_tp worker waiting -4967> 2014-05-19 18:36:55.342165 7fd0436bc700 20 OSD::op_tp worker waiting -4966> 2014-05-19 18:36:55.342292 7fd03bead700 20 OSD::op_tp worker waiting -4965> 2014-05-19 18:36:55.342483 7fd03f6b4700 20 OSD::op_tp worker waiting -4964> 2014-05-19 18:36:55.356594 7fd0446be700 20 OSD::op_tp worker waiting -4963> 2014-05-19 18:36:55.356620 7fd03feb5700 20 OSD::op_tp worker waiting -4962> 2014-05-19 18:36:55.363556 7fd040eb7700 20 OSD::op_tp worker waiting -4961> 2014-05-19 18:36:55.368961 7fd0406b6700 20 OSD::op_tp worker waiting -4960> 2014-05-19 18:36:55.390470 7fd039ea9700 20 OSD::disk_tp worker waiting -4959> 2014-05-19 18:36:55.394021 7fd03aeab700 20 OSD::op_tp worker waiting -4958> 2014-05-19 18:36:55.394068 7fd03eeb3700 20 OSD::op_tp worker waiting -4957> 2014-05-19 18:36:55.412059 7fd042ebb700 20 OSD::op_tp worker waiting -4956> 2014-05-19 18:36:55.412099 7fd03deb1700 20 OSD::op_tp worker waiting -4955> 2014-05-19 18:36:55.420631 7fd04e104700 20 KeyValueStore::op_tp worker waiting -4954> 2014-05-19 18:36:55.420634 7fd03b6ac700 20 OSD::op_tp worker waiting -4953> 2014-05-19 18:36:55.420664 7fd0426ba700 20 OSD::op_tp worker waiting -4952> 2014-05-19 18:36:55.420680 7fd03ceaf700 20 OSD::op_tp worker waiting -4951> 2014-05-19 18:36:55.420717 7fd0416b8700 20 OSD::op_tp worker waiting -4950> 2014-05-19 18:36:55.425559 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:58.482719) v2 ==== 47+0+0 (778343429 0 0) 0x5deb500 con 0x61ae420 -4949> 2014-05-19 18:36:55.425575 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:58.482719) v2 ==== 47+0+0 (778343429 0 0) 0x79e6540 con 0x618fe40 -4948> 2014-05-19 18:36:55.425606 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.482719) v2 -- ?+0 0x511c380 con 0x61ae420 -4947> 2014-05-19 18:36:55.425642 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4946> 2014-05-19 18:36:55.425663 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.482719) v2 -- ?+0 0x5e28380 con 0x618fe40 -4945> 2014-05-19 18:36:55.425699 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4944> 2014-05-19 18:36:55.493740 7fd04f907700 5 osd.0 21 tick -4943> 2014-05-19 18:36:55.493761 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4942> 2014-05-19 18:36:55.493765 7fd04f907700 10 osd.0 21 do_waiters -- start -4941> 2014-05-19 18:36:55.493768 7fd04f907700 10 osd.0 21 do_waiters -- finish -4940> 2014-05-19 18:36:55.493790 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4939> 2014-05-19 18:36:55.575329 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:55.575133) v2 ==== 47+0+0 (545310211 0 0) 0x612c8c0 con 0x4cb8160 -4938> 2014-05-19 18:36:55.575347 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:55.575133) v2 ==== 47+0+0 (545310211 0 0) 0x62e0540 con 0x4cb82c0 -4937> 2014-05-19 18:36:55.575370 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:55.575133) v2 -- ?+0 0x5deb500 con 0x4cb8160 -4936> 2014-05-19 18:36:55.575422 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4935> 2014-05-19 18:36:55.575462 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:55.575133) v2 -- ?+0 0x79e6540 con 0x4cb82c0 -4934> 2014-05-19 18:36:55.575497 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4933> 2014-05-19 18:36:55.614683 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:00.701790) v2 ==== 47+0+0 (2520255111 0 0) 0x6710e00 con 0x3094160 -4932> 2014-05-19 18:36:55.614708 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 72 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:00.701790) v2 ==== 47+0+0 (2520255111 0 0) 0x3438fc0 con 0x30946e0 -4931> 2014-05-19 18:36:55.614730 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.701790) v2 -- ?+0 0x62e0540 con 0x3094160 -4930> 2014-05-19 18:36:55.614778 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4929> 2014-05-19 18:36:55.614840 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.701790) v2 -- ?+0 0x612c8c0 con 0x30946e0 -4928> 2014-05-19 18:36:55.614876 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4927> 2014-05-19 18:36:55.794668 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:58.851823) v2 ==== 47+0+0 (913499 0 0) 0x6542fc0 con 0x61ae160 -4926> 2014-05-19 18:36:55.794671 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:58.851823) v2 ==== 47+0+0 (913499 0 0) 0x6c4d500 con 0x61b0420 -4925> 2014-05-19 18:36:55.794707 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.851823) v2 -- ?+0 0x3438fc0 con 0x61ae160 -4924> 2014-05-19 18:36:55.794738 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4923> 2014-05-19 18:36:55.794761 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.851823) v2 -- ?+0 0x6710e00 con 0x61b0420 -4922> 2014-05-19 18:36:55.794801 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4921> 2014-05-19 18:36:56.493819 7fd04f907700 5 osd.0 21 tick -4920> 2014-05-19 18:36:56.493839 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4919> 2014-05-19 18:36:56.493844 7fd04f907700 10 osd.0 21 do_waiters -- start -4918> 2014-05-19 18:36:56.493846 7fd04f907700 10 osd.0 21 do_waiters -- finish -4917> 2014-05-19 18:36:56.493887 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4916> 2014-05-19 18:36:56.675101 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:01.762217) v2 ==== 47+0+0 (3763957288 0 0) 0x72de380 con 0x31e1600 -4915> 2014-05-19 18:36:56.675154 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:01.762217) v2 -- ?+0 0x6542fc0 con 0x31e1600 -4914> 2014-05-19 18:36:56.675233 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4913> 2014-05-19 18:36:56.675111 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:01.762217) v2 ==== 47+0+0 (3763957288 0 0) 0x2f4fdc0 con 0x61b0160 -4912> 2014-05-19 18:36:56.675259 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:01.762217) v2 -- ?+0 0x6c4d500 con 0x61b0160 -4911> 2014-05-19 18:36:56.675290 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4910> 2014-05-19 18:36:56.775417 7fd04e905700 20 KeyValueStore::op_tp worker waiting -4909> 2014-05-19 18:36:56.775418 7fd043ebd700 20 OSD::op_tp worker waiting -4908> 2014-05-19 18:36:56.947710 7fd0396a8700 20 OSD::command_tp worker waiting -4907> 2014-05-19 18:36:57.341973 7fd041eb9700 20 OSD::op_tp worker waiting -4906> 2014-05-19 18:36:57.342027 7fd03e6b2700 20 OSD::op_tp worker waiting -4905> 2014-05-19 18:36:57.342050 7fd03c6ae700 20 OSD::op_tp worker waiting -4904> 2014-05-19 18:36:57.342077 7fd03d6b0700 20 OSD::op_tp worker waiting -4903> 2014-05-19 18:36:57.342231 7fd0436bc700 20 OSD::op_tp worker waiting -4902> 2014-05-19 18:36:57.342358 7fd03bead700 20 OSD::op_tp worker waiting -4901> 2014-05-19 18:36:57.342549 7fd03f6b4700 20 OSD::op_tp worker waiting -4900> 2014-05-19 18:36:57.356688 7fd0446be700 20 OSD::op_tp worker waiting -4899> 2014-05-19 18:36:57.356723 7fd03feb5700 20 OSD::op_tp worker waiting -4898> 2014-05-19 18:36:57.363620 7fd040eb7700 20 OSD::op_tp worker waiting -4897> 2014-05-19 18:36:57.369058 7fd0406b6700 20 OSD::op_tp worker waiting -4896> 2014-05-19 18:36:57.390566 7fd039ea9700 20 OSD::disk_tp worker waiting -4895> 2014-05-19 18:36:57.394107 7fd03aeab700 20 OSD::op_tp worker waiting -4894> 2014-05-19 18:36:57.394148 7fd03eeb3700 20 OSD::op_tp worker waiting -4893> 2014-05-19 18:36:57.412143 7fd042ebb700 20 OSD::op_tp worker waiting -4892> 2014-05-19 18:36:57.412180 7fd03deb1700 20 OSD::op_tp worker waiting -4891> 2014-05-19 18:36:57.420727 7fd04e104700 20 KeyValueStore::op_tp worker waiting -4890> 2014-05-19 18:36:57.420732 7fd03ceaf700 20 OSD::op_tp worker waiting -4889> 2014-05-19 18:36:57.420752 7fd03b6ac700 20 OSD::op_tp worker waiting -4888> 2014-05-19 18:36:57.420784 7fd0426ba700 20 OSD::op_tp worker waiting -4887> 2014-05-19 18:36:57.420809 7fd0416b8700 20 OSD::op_tp worker waiting -4886> 2014-05-19 18:36:57.490151 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:57.489967) v2 ==== 47+0+0 (3878068376 0 0) 0x8648e00 con 0x4cb86e0 -4885> 2014-05-19 18:36:57.490159 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:36:57.489967) v2 ==== 47+0+0 (3878068376 0 0) 0x799a700 con 0x4cb8420 -4884> 2014-05-19 18:36:57.490214 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:57.489967) v2 -- ?+0 0x72de380 con 0x4cb86e0 -4883> 2014-05-19 18:36:57.490269 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4882> 2014-05-19 18:36:57.490316 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:36:57.489967) v2 -- ?+0 0x2f4fdc0 con 0x4cb8420 -4881> 2014-05-19 18:36:57.490357 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4880> 2014-05-19 18:36:57.493917 7fd04f907700 5 osd.0 21 tick -4879> 2014-05-19 18:36:57.493935 7fd04f907700 7 osd.0 21 do_mon_report -4878> 2014-05-19 18:36:57.493939 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -4877> 2014-05-19 18:36:57.493942 7fd04f907700 20 osd.0 21 send_pg_stats -4876> 2014-05-19 18:36:57.493951 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4875> 2014-05-19 18:36:57.493954 7fd04f907700 10 osd.0 21 do_waiters -- start -4874> 2014-05-19 18:36:57.493956 7fd04f907700 10 osd.0 21 do_waiters -- finish -4873> 2014-05-19 18:36:57.493968 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4872> 2014-05-19 18:36:58.494040 7fd04f907700 5 osd.0 21 tick -4871> 2014-05-19 18:36:58.494061 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4870> 2014-05-19 18:36:58.494065 7fd04f907700 10 osd.0 21 do_waiters -- start -4869> 2014-05-19 18:36:58.494068 7fd04f907700 10 osd.0 21 do_waiters -- finish -4868> 2014-05-19 18:36:58.494091 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4867> 2014-05-19 18:36:58.564109 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:01.621327) v2 ==== 47+0+0 (2127866052 0 0) 0x5beb340 con 0x4cb8dc0 -4866> 2014-05-19 18:36:58.564111 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:01.621327) v2 ==== 47+0+0 (2127866052 0 0) 0x68aae00 con 0x4cb9340 -4865> 2014-05-19 18:36:58.564164 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:01.621327) v2 -- ?+0 0x8648e00 con 0x4cb8dc0 -4864> 2014-05-19 18:36:58.564202 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4863> 2014-05-19 18:36:58.564246 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:01.621327) v2 -- ?+0 0x799a700 con 0x4cb9340 -4862> 2014-05-19 18:36:58.564281 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4861> 2014-05-19 18:36:58.775510 7fd04e905700 20 KeyValueStore::op_tp worker waiting -4860> 2014-05-19 18:36:58.775515 7fd043ebd700 20 OSD::op_tp worker waiting -4859> 2014-05-19 18:36:58.947782 7fd0396a8700 20 OSD::command_tp worker waiting -4858> 2014-05-19 18:36:58.960185 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -4857> 2014-05-19 18:36:58.960204 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -4856> 2014-05-19 18:36:58.960212 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x6b70380 con 0x3622580 -4855> 2014-05-19 18:36:58.960237 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x682e700 con 0x3622840 -4854> 2014-05-19 18:36:58.960277 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x6d61500 con 0x3622b00 -4853> 2014-05-19 18:36:58.960291 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x7998a80 con 0x3622dc0 -4852> 2014-05-19 18:36:58.960319 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x798e380 con 0x30594a0 -4851> 2014-05-19 18:36:58.960338 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x72e4540 con 0x3622f20 -4850> 2014-05-19 18:36:58.960357 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x77336c0 con 0x5e7ef20 -4849> 2014-05-19 18:36:58.960388 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x6e19c00 con 0x5e7f1e0 -4848> 2014-05-19 18:36:58.960417 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x70a9dc0 con 0x5e7e2c0 -4847> 2014-05-19 18:36:58.960443 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x78b8c40 con 0x5d7bb80 -4846> 2014-05-19 18:36:58.960468 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x72e5dc0 con 0x5e7ec60 -4845> 2014-05-19 18:36:58.960490 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x64d2a80 con 0x531b760 -4844> 2014-05-19 18:36:58.960518 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x5f30e00 con 0x53bba20 -4843> 2014-05-19 18:36:58.960547 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x784d880 con 0x51cac60 -4842> 2014-05-19 18:36:58.960597 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x662ba40 con 0x5bd66e0 -4841> 2014-05-19 18:36:58.960621 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:36:58.960210) v2 -- ?+0 0x5cbcfc0 con 0x5bd6840 -4840> 2014-05-19 18:36:58.960695 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x5e288c0 con 0x3622840 -4839> 2014-05-19 18:36:58.960786 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4838> 2014-05-19 18:36:58.960803 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x5b28380 con 0x3622580 -4837> 2014-05-19 18:36:58.960823 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4836> 2014-05-19 18:36:58.960832 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x6710fc0 con 0x3622b00 -4835> 2014-05-19 18:36:58.960847 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4834> 2014-05-19 18:36:58.960853 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x7930700 con 0x3622dc0 -4833> 2014-05-19 18:36:58.960867 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4832> 2014-05-19 18:36:58.960873 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x6c4d6c0 con 0x5e7ef20 -4831> 2014-05-19 18:36:58.960887 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4830> 2014-05-19 18:36:58.960894 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x692d180 con 0x30594a0 -4829> 2014-05-19 18:36:58.960909 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4828> 2014-05-19 18:36:58.960922 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x7746c40 con 0x3622f20 -4827> 2014-05-19 18:36:58.960945 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4826> 2014-05-19 18:36:58.960953 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x73c21c0 con 0x5e7f1e0 -4825> 2014-05-19 18:36:58.960970 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4824> 2014-05-19 18:36:58.960980 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x860f6c0 con 0x5e7e2c0 -4823> 2014-05-19 18:36:58.961019 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4822> 2014-05-19 18:36:58.961048 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x7980380 con 0x5d7bb80 -4821> 2014-05-19 18:36:58.961066 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4820> 2014-05-19 18:36:58.961082 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x6009c00 con 0x531b760 -4819> 2014-05-19 18:36:58.961107 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4818> 2014-05-19 18:36:58.961114 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x5dff180 con 0x53bba20 -4817> 2014-05-19 18:36:58.961132 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4816> 2014-05-19 18:36:58.961142 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x73c2e00 con 0x51cac60 -4815> 2014-05-19 18:36:58.961157 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4814> 2014-05-19 18:36:58.961165 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x5666a80 con 0x5bd66e0 -4813> 2014-05-19 18:36:58.961179 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4812> 2014-05-19 18:36:58.961187 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 77 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x649e000 con 0x5bd6840 -4811> 2014-05-19 18:36:58.961200 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4810> 2014-05-19 18:36:58.961208 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:36:58.960210) v2 ==== 47+0+0 (1992004094 0 0) 0x5c05880 con 0x5e7ec60 -4809> 2014-05-19 18:36:58.961222 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4808> 2014-05-19 18:36:59.342069 7fd041eb9700 20 OSD::op_tp worker waiting -4807> 2014-05-19 18:36:59.342125 7fd03e6b2700 20 OSD::op_tp worker waiting -4806> 2014-05-19 18:36:59.342145 7fd03d6b0700 20 OSD::op_tp worker waiting -4805> 2014-05-19 18:36:59.342159 7fd03c6ae700 20 OSD::op_tp worker waiting -4804> 2014-05-19 18:36:59.342299 7fd0436bc700 20 OSD::op_tp worker waiting -4803> 2014-05-19 18:36:59.342418 7fd03bead700 20 OSD::op_tp worker waiting -4802> 2014-05-19 18:36:59.342615 7fd03f6b4700 20 OSD::op_tp worker waiting -4801> 2014-05-19 18:36:59.356790 7fd0446be700 20 OSD::op_tp worker waiting -4800> 2014-05-19 18:36:59.356824 7fd03feb5700 20 OSD::op_tp worker waiting -4799> 2014-05-19 18:36:59.363711 7fd040eb7700 20 OSD::op_tp worker waiting -4798> 2014-05-19 18:36:59.369154 7fd0406b6700 20 OSD::op_tp worker waiting -4797> 2014-05-19 18:36:59.390661 7fd039ea9700 20 OSD::disk_tp worker waiting -4796> 2014-05-19 18:36:59.394206 7fd03aeab700 20 OSD::op_tp worker waiting -4795> 2014-05-19 18:36:59.394228 7fd03eeb3700 20 OSD::op_tp worker waiting -4794> 2014-05-19 18:36:59.412227 7fd042ebb700 20 OSD::op_tp worker waiting -4793> 2014-05-19 18:36:59.412258 7fd03deb1700 20 OSD::op_tp worker waiting -4792> 2014-05-19 18:36:59.420825 7fd04e104700 20 KeyValueStore::op_tp worker waiting -4791> 2014-05-19 18:36:59.420829 7fd03ceaf700 20 OSD::op_tp worker waiting -4790> 2014-05-19 18:36:59.420863 7fd03b6ac700 20 OSD::op_tp worker waiting -4789> 2014-05-19 18:36:59.420889 7fd0426ba700 20 OSD::op_tp worker waiting -4788> 2014-05-19 18:36:59.420939 7fd0416b8700 20 OSD::op_tp worker waiting -4787> 2014-05-19 18:36:59.494152 7fd04f907700 5 osd.0 21 tick -4786> 2014-05-19 18:36:59.494200 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.04 < max 0.5 = yes -4785> 2014-05-19 18:36:59.494213 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -4784> 2014-05-19 18:36:59.494220 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4783> 2014-05-19 18:36:59.494216 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 262.218 < min (86400 seconds) -4782> 2014-05-19 18:36:59.494234 7fd04f907700 20 osd.0 21 sched_scrub done -4781> 2014-05-19 18:36:59.494237 7fd04f907700 10 osd.0 21 do_waiters -- start -4780> 2014-05-19 18:36:59.494239 7fd04f907700 10 osd.0 21 do_waiters -- finish -4779> 2014-05-19 18:36:59.907501 7fd0456c0700 10 monclient: tick -4778> 2014-05-19 18:36:59.907513 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:36:29.907512) -4777> 2014-05-19 18:36:59.907530 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:36:59.907530; renew after: 2014-05-19 18:38:19.906664) -- no -4776> 2014-05-19 18:37:00.060751 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -4775> 2014-05-19 18:37:00.060769 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -4774> 2014-05-19 18:37:00.060777 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x6009180 con 0x3622580 -4773> 2014-05-19 18:37:00.060803 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x6b71340 con 0x3622840 -4772> 2014-05-19 18:37:00.060841 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x3511a40 con 0x3622b00 -4771> 2014-05-19 18:37:00.060856 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x6d60700 con 0x3622dc0 -4770> 2014-05-19 18:37:00.060869 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x5f7b180 con 0x30594a0 -4769> 2014-05-19 18:37:00.060883 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x79716c0 con 0x3622f20 -4768> 2014-05-19 18:37:00.060902 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x692d340 con 0x5e7ef20 -4767> 2014-05-19 18:37:00.060918 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x74a6700 con 0x5e7f1e0 -4766> 2014-05-19 18:37:00.060942 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x5c04e00 con 0x5e7e2c0 -4765> 2014-05-19 18:37:00.060968 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x5dea8c0 con 0x5d7bb80 -4764> 2014-05-19 18:37:00.060993 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x6b716c0 con 0x5e7ec60 -4763> 2014-05-19 18:37:00.061017 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x73c0000 con 0x531b760 -4762> 2014-05-19 18:37:00.061041 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x72e5500 con 0x53bba20 -4761> 2014-05-19 18:37:00.061065 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x76acfc0 con 0x51cac60 -4760> 2014-05-19 18:37:00.061087 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x5f30540 con 0x5bd66e0 -4759> 2014-05-19 18:37:00.061110 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:00.060775) v2 -- ?+0 0x5666c40 con 0x5bd6840 -4758> 2014-05-19 18:37:00.061198 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x6fe61c0 con 0x3622dc0 -4757> 2014-05-19 18:37:00.061262 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4756> 2014-05-19 18:37:00.061273 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x5e29340 con 0x3622840 -4755> 2014-05-19 18:37:00.061299 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4754> 2014-05-19 18:37:00.061320 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x692da40 con 0x30594a0 -4753> 2014-05-19 18:37:00.061347 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4752> 2014-05-19 18:37:00.061436 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x5b28540 con 0x3622580 -4751> 2014-05-19 18:37:00.061483 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4750> 2014-05-19 18:37:00.061496 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x6711180 con 0x3622b00 -4749> 2014-05-19 18:37:00.061526 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4748> 2014-05-19 18:37:00.061537 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x6c4cc40 con 0x5e7ef20 -4747> 2014-05-19 18:37:00.061589 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4746> 2014-05-19 18:37:00.061603 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x73c2700 con 0x5e7f1e0 -4745> 2014-05-19 18:37:00.061628 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4744> 2014-05-19 18:37:00.061637 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x59bc540 con 0x5e7e2c0 -4743> 2014-05-19 18:37:00.061654 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4742> 2014-05-19 18:37:00.061662 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x5667c00 con 0x5d7bb80 -4741> 2014-05-19 18:37:00.061677 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4740> 2014-05-19 18:37:00.061682 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x662b880 con 0x5e7ec60 -4739> 2014-05-19 18:37:00.061698 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4738> 2014-05-19 18:37:00.061708 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x6008380 con 0x531b760 -4737> 2014-05-19 18:37:00.061723 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4736> 2014-05-19 18:37:00.061730 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x66f9a40 con 0x53bba20 -4735> 2014-05-19 18:37:00.061745 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4734> 2014-05-19 18:37:00.061754 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x6542000 con 0x51cac60 -4733> 2014-05-19 18:37:00.061771 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4732> 2014-05-19 18:37:00.061778 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x79801c0 con 0x5bd66e0 -4731> 2014-05-19 18:37:00.061793 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4730> 2014-05-19 18:37:00.061800 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 78 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x649ea80 con 0x5bd6840 -4729> 2014-05-19 18:37:00.061814 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4728> 2014-05-19 18:37:00.061822 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.060775) v2 ==== 47+0+0 (2518493763 0 0) 0x7747880 con 0x3622f20 -4727> 2014-05-19 18:37:00.061837 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4726> 2014-05-19 18:37:00.494310 7fd04f907700 5 osd.0 21 tick -4725> 2014-05-19 18:37:00.494332 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4724> 2014-05-19 18:37:00.494336 7fd04f907700 10 osd.0 21 do_waiters -- start -4723> 2014-05-19 18:37:00.494339 7fd04f907700 10 osd.0 21 do_waiters -- finish -4722> 2014-05-19 18:37:00.494362 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4721> 2014-05-19 18:37:00.495113 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:03.552317) v2 ==== 47+0+0 (3387442200 0 0) 0x70ea380 con 0x61b0420 -4720> 2014-05-19 18:37:00.495158 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.552317) v2 -- ?+0 0x68aae00 con 0x61b0420 -4719> 2014-05-19 18:37:00.495190 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4718> 2014-05-19 18:37:00.495200 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:03.552317) v2 ==== 47+0+0 (3387442200 0 0) 0x72e4c40 con 0x61ae160 -4717> 2014-05-19 18:37:00.495231 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.552317) v2 -- ?+0 0x5beb340 con 0x61ae160 -4716> 2014-05-19 18:37:00.495256 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4715> 2014-05-19 18:37:00.590438 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:05.677663) v2 ==== 47+0+0 (1613162932 0 0) 0x5497340 con 0x5f5b760 -4714> 2014-05-19 18:37:00.590421 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:05.677663) v2 ==== 47+0+0 (1613162932 0 0) 0x691a700 con 0x5b21600 -4713> 2014-05-19 18:37:00.590482 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:05.677663) v2 -- ?+0 0x70ea380 con 0x5f5b760 -4712> 2014-05-19 18:37:00.590518 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4711> 2014-05-19 18:37:00.590542 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:05.677663) v2 -- ?+0 0x72e4c40 con 0x5b21600 -4710> 2014-05-19 18:37:00.590618 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4709> 2014-05-19 18:37:00.726021 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:03.783246) v2 ==== 47+0+0 (2525090930 0 0) 0x5bea540 con 0x61ae420 -4708> 2014-05-19 18:37:00.726046 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:03.783246) v2 ==== 47+0+0 (2525090930 0 0) 0x79e68c0 con 0x618fe40 -4707> 2014-05-19 18:37:00.726069 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.783246) v2 -- ?+0 0x691a700 con 0x61ae420 -4706> 2014-05-19 18:37:00.726125 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4705> 2014-05-19 18:37:00.726147 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.783246) v2 -- ?+0 0x5497340 con 0x618fe40 -4704> 2014-05-19 18:37:00.726183 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4703> 2014-05-19 18:37:00.775602 7fd04e905700 20 KeyValueStore::op_tp worker waiting -4702> 2014-05-19 18:37:00.775604 7fd043ebd700 20 OSD::op_tp worker waiting -4701> 2014-05-19 18:37:00.875859 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:00.875686) v2 ==== 47+0+0 (62630125 0 0) 0x70cda40 con 0x4cb8160 -4700> 2014-05-19 18:37:00.875898 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.875686) v2 -- ?+0 0x5bea540 con 0x4cb8160 -4699> 2014-05-19 18:37:00.875888 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:00.875686) v2 ==== 47+0+0 (62630125 0 0) 0x63d8c40 con 0x4cb82c0 -4698> 2014-05-19 18:37:00.875926 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4697> 2014-05-19 18:37:00.875937 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:00.875686) v2 -- ?+0 0x79e68c0 con 0x4cb82c0 -4696> 2014-05-19 18:37:00.875989 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4695> 2014-05-19 18:37:00.947863 7fd0396a8700 20 OSD::command_tp worker waiting -4694> 2014-05-19 18:37:01.342169 7fd041eb9700 20 OSD::op_tp worker waiting -4693> 2014-05-19 18:37:01.342203 7fd03d6b0700 20 OSD::op_tp worker waiting -4692> 2014-05-19 18:37:01.342227 7fd03e6b2700 20 OSD::op_tp worker waiting -4691> 2014-05-19 18:37:01.342252 7fd03c6ae700 20 OSD::op_tp worker waiting -4690> 2014-05-19 18:37:01.342369 7fd0436bc700 20 OSD::op_tp worker waiting -4689> 2014-05-19 18:37:01.342484 7fd03bead700 20 OSD::op_tp worker waiting -4688> 2014-05-19 18:37:01.342692 7fd03f6b4700 20 OSD::op_tp worker waiting -4687> 2014-05-19 18:37:01.356894 7fd0446be700 20 OSD::op_tp worker waiting -4686> 2014-05-19 18:37:01.356925 7fd03feb5700 20 OSD::op_tp worker waiting -4685> 2014-05-19 18:37:01.363794 7fd040eb7700 20 OSD::op_tp worker waiting -4684> 2014-05-19 18:37:01.369235 7fd0406b6700 20 OSD::op_tp worker waiting -4683> 2014-05-19 18:37:01.375470 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:06.462754) v2 ==== 47+0+0 (2727314696 0 0) 0x51a7dc0 con 0x31e1600 -4682> 2014-05-19 18:37:01.375498 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:06.462754) v2 ==== 47+0+0 (2727314696 0 0) 0x711e700 con 0x61b0160 -4681> 2014-05-19 18:37:01.375525 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.462754) v2 -- ?+0 0x70cda40 con 0x31e1600 -4680> 2014-05-19 18:37:01.375561 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4679> 2014-05-19 18:37:01.375584 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.462754) v2 -- ?+0 0x63d8c40 con 0x61b0160 -4678> 2014-05-19 18:37:01.375623 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4677> 2014-05-19 18:37:01.390757 7fd039ea9700 20 OSD::disk_tp worker waiting -4676> 2014-05-19 18:37:01.394307 7fd03aeab700 20 OSD::op_tp worker waiting -4675> 2014-05-19 18:37:01.394368 7fd03eeb3700 20 OSD::op_tp worker waiting -4674> 2014-05-19 18:37:01.412308 7fd042ebb700 20 OSD::op_tp worker waiting -4673> 2014-05-19 18:37:01.412338 7fd03deb1700 20 OSD::op_tp worker waiting -4672> 2014-05-19 18:37:01.420918 7fd04e104700 20 KeyValueStore::op_tp worker waiting -4671> 2014-05-19 18:37:01.420924 7fd03ceaf700 20 OSD::op_tp worker waiting -4670> 2014-05-19 18:37:01.420943 7fd03b6ac700 20 OSD::op_tp worker waiting -4669> 2014-05-19 18:37:01.420967 7fd0426ba700 20 OSD::op_tp worker waiting -4668> 2014-05-19 18:37:01.421015 7fd0416b8700 20 OSD::op_tp worker waiting -4667> 2014-05-19 18:37:01.494423 7fd04f907700 5 osd.0 21 tick -4666> 2014-05-19 18:37:01.494471 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.04 < max 0.5 = yes -4665> 2014-05-19 18:37:01.494485 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -4664> 2014-05-19 18:37:01.494492 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4663> 2014-05-19 18:37:01.494488 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 264.219 < min (86400 seconds) -4662> 2014-05-19 18:37:01.494507 7fd04f907700 20 osd.0 21 sched_scrub done -4661> 2014-05-19 18:37:01.494510 7fd04f907700 10 osd.0 21 do_waiters -- start -4660> 2014-05-19 18:37:01.494512 7fd04f907700 10 osd.0 21 do_waiters -- finish -4659> 2014-05-19 18:37:01.514991 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:06.602286) v2 ==== 47+0+0 (178143731 0 0) 0x5519c00 con 0x30946e0 -4658> 2014-05-19 18:37:01.515011 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 73 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:06.602286) v2 ==== 47+0+0 (178143731 0 0) 0x5497dc0 con 0x3094160 -4657> 2014-05-19 18:37:01.515039 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.602286) v2 -- ?+0 0x51a7dc0 con 0x30946e0 -4656> 2014-05-19 18:37:01.515079 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4655> 2014-05-19 18:37:01.515102 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.602286) v2 -- ?+0 0x711e700 con 0x3094160 -4654> 2014-05-19 18:37:01.515144 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4653> 2014-05-19 18:37:01.826528 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:04.883754) v2 ==== 47+0+0 (2222543823 0 0) 0x5beba40 con 0x61ae420 -4652> 2014-05-19 18:37:01.826519 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:04.883754) v2 ==== 47+0+0 (2222543823 0 0) 0x5cbc540 con 0x618fe40 -4651> 2014-05-19 18:37:01.826572 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:04.883754) v2 -- ?+0 0x5519c00 con 0x61ae420 -4650> 2014-05-19 18:37:01.826608 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4649> 2014-05-19 18:37:01.826666 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:04.883754) v2 -- ?+0 0x5497dc0 con 0x618fe40 -4648> 2014-05-19 18:37:01.826710 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4647> 2014-05-19 18:37:01.976400 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:01.976226) v2 ==== 47+0+0 (876538738 0 0) 0x64ae700 con 0x4cb82c0 -4646> 2014-05-19 18:37:01.976412 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:01.976226) v2 ==== 47+0+0 (876538738 0 0) 0x70cc1c0 con 0x4cb8160 -4645> 2014-05-19 18:37:01.976448 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:01.976226) v2 -- ?+0 0x5cbc540 con 0x4cb82c0 -4644> 2014-05-19 18:37:01.976500 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4643> 2014-05-19 18:37:01.976609 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:01.976226) v2 -- ?+0 0x5beba40 con 0x4cb8160 -4642> 2014-05-19 18:37:01.976714 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4641> 2014-05-19 18:37:02.494595 7fd04f907700 5 osd.0 21 tick -4640> 2014-05-19 18:37:02.494612 7fd04f907700 7 osd.0 21 do_mon_report -4639> 2014-05-19 18:37:02.494616 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -4638> 2014-05-19 18:37:02.494619 7fd04f907700 20 osd.0 21 send_pg_stats -4637> 2014-05-19 18:37:02.494627 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4636> 2014-05-19 18:37:02.494630 7fd04f907700 10 osd.0 21 do_waiters -- start -4635> 2014-05-19 18:37:02.494633 7fd04f907700 10 osd.0 21 do_waiters -- finish -4634> 2014-05-19 18:37:02.494671 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4633> 2014-05-19 18:37:02.775673 7fd04e905700 20 KeyValueStore::op_tp worker waiting -4632> 2014-05-19 18:37:02.775695 7fd043ebd700 20 OSD::op_tp worker waiting -4631> 2014-05-19 18:37:02.790708 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:02.790520) v2 ==== 47+0+0 (1601303047 0 0) 0x8649a40 con 0x4cb86e0 -4630> 2014-05-19 18:37:02.790757 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.790520) v2 -- ?+0 0x70cc1c0 con 0x4cb86e0 -4629> 2014-05-19 18:37:02.790796 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4628> 2014-05-19 18:37:02.790756 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:02.790520) v2 ==== 47+0+0 (1601303047 0 0) 0x5e6ca80 con 0x4cb8420 -4627> 2014-05-19 18:37:02.790878 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.790520) v2 -- ?+0 0x64ae700 con 0x4cb8420 -4626> 2014-05-19 18:37:02.790955 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4625> 2014-05-19 18:37:02.947932 7fd0396a8700 20 OSD::command_tp worker waiting -4624> 2014-05-19 18:37:02.961252 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -4623> 2014-05-19 18:37:02.961270 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -4622> 2014-05-19 18:37:02.961278 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x503e000 con 0x3622580 -4621> 2014-05-19 18:37:02.961303 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x691b6c0 con 0x3622840 -4620> 2014-05-19 18:37:02.961342 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x6008540 con 0x3622b00 -4619> 2014-05-19 18:37:02.961356 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x51a68c0 con 0x3622dc0 -4618> 2014-05-19 18:37:02.961370 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x711fa40 con 0x30594a0 -4617> 2014-05-19 18:37:02.961383 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x5ccc8c0 con 0x3622f20 -4616> 2014-05-19 18:37:02.961405 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x7970e00 con 0x5e7ef20 -4615> 2014-05-19 18:37:02.961426 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x692d500 con 0x5e7f1e0 -4614> 2014-05-19 18:37:02.961449 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x6711500 con 0x5e7e2c0 -4613> 2014-05-19 18:37:02.961475 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x78b9a40 con 0x5d7bb80 -4612> 2014-05-19 18:37:02.961502 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x5f31880 con 0x5e7ec60 -4611> 2014-05-19 18:37:02.961527 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x5667a40 con 0x531b760 -4610> 2014-05-19 18:37:02.961557 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x75b1c00 con 0x53bba20 -4609> 2014-05-19 18:37:02.961582 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x70a88c0 con 0x51cac60 -4608> 2014-05-19 18:37:02.961611 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x6008000 con 0x5bd66e0 -4607> 2014-05-19 18:37:02.961635 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:02.961276) v2 -- ?+0 0x6b70540 con 0x5bd6840 -4606> 2014-05-19 18:37:02.961669 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x5e28e00 con 0x3622580 -4605> 2014-05-19 18:37:02.961744 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4604> 2014-05-19 18:37:02.961765 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x6fe7880 con 0x3622dc0 -4603> 2014-05-19 18:37:02.961789 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4602> 2014-05-19 18:37:02.961800 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x60a0540 con 0x3622840 -4601> 2014-05-19 18:37:02.961815 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4600> 2014-05-19 18:37:02.961834 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x67101c0 con 0x3622b00 -4599> 2014-05-19 18:37:02.961857 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4598> 2014-05-19 18:37:02.961863 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x6c4c540 con 0x5e7ef20 -4597> 2014-05-19 18:37:02.961906 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4596> 2014-05-19 18:37:02.961919 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x5667340 con 0x5d7bb80 -4595> 2014-05-19 18:37:02.961948 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4594> 2014-05-19 18:37:02.961982 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x692c000 con 0x30594a0 -4593> 2014-05-19 18:37:02.962004 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4592> 2014-05-19 18:37:02.962027 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x7733340 con 0x3622f20 -4591> 2014-05-19 18:37:02.962054 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4590> 2014-05-19 18:37:02.962060 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x6c4a000 con 0x5e7f1e0 -4589> 2014-05-19 18:37:02.962075 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4588> 2014-05-19 18:37:02.962080 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x59bc1c0 con 0x5e7e2c0 -4587> 2014-05-19 18:37:02.962094 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4586> 2014-05-19 18:37:02.962103 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x662b180 con 0x5e7ec60 -4585> 2014-05-19 18:37:02.962117 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4584> 2014-05-19 18:37:02.962128 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x6009a40 con 0x531b760 -4583> 2014-05-19 18:37:02.962142 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4582> 2014-05-19 18:37:02.962152 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x66f8380 con 0x53bba20 -4581> 2014-05-19 18:37:02.962169 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4580> 2014-05-19 18:37:02.962177 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x6542c40 con 0x51cac60 -4579> 2014-05-19 18:37:02.962191 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4578> 2014-05-19 18:37:02.962199 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x7aa4fc0 con 0x5bd66e0 -4577> 2014-05-19 18:37:02.962213 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4576> 2014-05-19 18:37:02.962219 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 79 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:02.961276) v2 ==== 47+0+0 (3075766436 0 0) 0x656ca80 con 0x5bd6840 -4575> 2014-05-19 18:37:02.962236 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4574> 2014-05-19 18:37:03.342263 7fd041eb9700 20 OSD::op_tp worker waiting -4573> 2014-05-19 18:37:03.342284 7fd03d6b0700 20 OSD::op_tp worker waiting -4572> 2014-05-19 18:37:03.342295 7fd03e6b2700 20 OSD::op_tp worker waiting -4571> 2014-05-19 18:37:03.342323 7fd03c6ae700 20 OSD::op_tp worker waiting -4570> 2014-05-19 18:37:03.342433 7fd0436bc700 20 OSD::op_tp worker waiting -4569> 2014-05-19 18:37:03.342549 7fd03bead700 20 OSD::op_tp worker waiting -4568> 2014-05-19 18:37:03.342765 7fd03f6b4700 20 OSD::op_tp worker waiting -4567> 2014-05-19 18:37:03.356987 7fd0446be700 20 OSD::op_tp worker waiting -4566> 2014-05-19 18:37:03.357020 7fd03feb5700 20 OSD::op_tp worker waiting -4565> 2014-05-19 18:37:03.363875 7fd040eb7700 20 OSD::op_tp worker waiting -4564> 2014-05-19 18:37:03.369331 7fd0406b6700 20 OSD::op_tp worker waiting -4563> 2014-05-19 18:37:03.390854 7fd039ea9700 20 OSD::disk_tp worker waiting -4562> 2014-05-19 18:37:03.394406 7fd03aeab700 20 OSD::op_tp worker waiting -4561> 2014-05-19 18:37:03.394455 7fd03eeb3700 20 OSD::op_tp worker waiting -4560> 2014-05-19 18:37:03.412394 7fd042ebb700 20 OSD::op_tp worker waiting -4559> 2014-05-19 18:37:03.412416 7fd03deb1700 20 OSD::op_tp worker waiting -4558> 2014-05-19 18:37:03.421018 7fd04e104700 20 KeyValueStore::op_tp worker waiting -4557> 2014-05-19 18:37:03.421023 7fd03ceaf700 20 OSD::op_tp worker waiting -4556> 2014-05-19 18:37:03.421106 7fd03b6ac700 20 OSD::op_tp worker waiting -4555> 2014-05-19 18:37:03.421156 7fd0426ba700 20 OSD::op_tp worker waiting -4554> 2014-05-19 18:37:03.421215 7fd0416b8700 20 OSD::op_tp worker waiting -4553> 2014-05-19 18:37:03.461848 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -4552> 2014-05-19 18:37:03.461867 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -4551> 2014-05-19 18:37:03.461874 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x682f340 con 0x3622580 -4550> 2014-05-19 18:37:03.461900 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x5ccc1c0 con 0x3622840 -4549> 2014-05-19 18:37:03.461938 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x7732540 con 0x3622b00 -4548> 2014-05-19 18:37:03.461952 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x5f0b880 con 0x3622dc0 -4547> 2014-05-19 18:37:03.461966 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x6c4cc40 con 0x30594a0 -4546> 2014-05-19 18:37:03.461980 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x6711180 con 0x3622f20 -4545> 2014-05-19 18:37:03.462008 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x5b28540 con 0x5e7ef20 -4544> 2014-05-19 18:37:03.462034 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x692da40 con 0x5e7f1e0 -4543> 2014-05-19 18:37:03.462060 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x5e29340 con 0x5e7e2c0 -4542> 2014-05-19 18:37:03.462086 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x6fe61c0 con 0x5d7bb80 -4541> 2014-05-19 18:37:03.462110 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x5c05880 con 0x5e7ec60 -4540> 2014-05-19 18:37:03.462133 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x649e000 con 0x531b760 -4539> 2014-05-19 18:37:03.462158 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x5666a80 con 0x53bba20 -4538> 2014-05-19 18:37:03.462183 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x73c2e00 con 0x51cac60 -4537> 2014-05-19 18:37:03.462208 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x5dff180 con 0x5bd66e0 -4536> 2014-05-19 18:37:03.462235 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:03.461873) v2 -- ?+0 0x6009c00 con 0x5bd6840 -4535> 2014-05-19 18:37:03.462268 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x60a1a40 con 0x3622840 -4534> 2014-05-19 18:37:03.462332 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4533> 2014-05-19 18:37:03.462350 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x725da40 con 0x3622b00 -4532> 2014-05-19 18:37:03.462382 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4531> 2014-05-19 18:37:03.462404 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x5e28c40 con 0x3622580 -4530> 2014-05-19 18:37:03.462426 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4529> 2014-05-19 18:37:03.462445 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x6fe7a40 con 0x3622dc0 -4528> 2014-05-19 18:37:03.462461 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4527> 2014-05-19 18:37:03.462482 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x59bcfc0 con 0x5e7e2c0 -4526> 2014-05-19 18:37:03.462505 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4525> 2014-05-19 18:37:03.462511 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x6c4c000 con 0x5e7ef20 -4524> 2014-05-19 18:37:03.462554 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4523> 2014-05-19 18:37:03.462596 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x692dc00 con 0x30594a0 -4522> 2014-05-19 18:37:03.462625 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4521> 2014-05-19 18:37:03.462642 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x5f0a380 con 0x5d7bb80 -4520> 2014-05-19 18:37:03.462668 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4519> 2014-05-19 18:37:03.462685 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x60081c0 con 0x531b760 -4518> 2014-05-19 18:37:03.462717 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4517> 2014-05-19 18:37:03.462730 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x66f9dc0 con 0x53bba20 -4516> 2014-05-19 18:37:03.462763 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4515> 2014-05-19 18:37:03.462780 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x6543500 con 0x51cac60 -4514> 2014-05-19 18:37:03.462813 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4513> 2014-05-19 18:37:03.462829 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x7aa4540 con 0x5bd66e0 -4512> 2014-05-19 18:37:03.462859 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4511> 2014-05-19 18:37:03.462875 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 80 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x656cfc0 con 0x5bd6840 -4510> 2014-05-19 18:37:03.462906 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4509> 2014-05-19 18:37:03.462919 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x7733a40 con 0x3622f20 -4508> 2014-05-19 18:37:03.462949 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4507> 2014-05-19 18:37:03.462962 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x6c4b180 con 0x5e7f1e0 -4506> 2014-05-19 18:37:03.462991 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4505> 2014-05-19 18:37:03.463004 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.461873) v2 ==== 47+0+0 (1016292603 0 0) 0x662b6c0 con 0x5e7ec60 -4504> 2014-05-19 18:37:03.463030 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4503> 2014-05-19 18:37:03.494703 7fd04f907700 5 osd.0 21 tick -4502> 2014-05-19 18:37:03.494752 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.04 < max 0.5 = yes -4501> 2014-05-19 18:37:03.494763 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4500> 2014-05-19 18:37:03.494768 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -4499> 2014-05-19 18:37:03.494771 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 266.219 < min (86400 seconds) -4498> 2014-05-19 18:37:03.494790 7fd04f907700 20 osd.0 21 sched_scrub done -4497> 2014-05-19 18:37:03.494793 7fd04f907700 10 osd.0 21 do_waiters -- start -4496> 2014-05-19 18:37:03.494794 7fd04f907700 10 osd.0 21 do_waiters -- finish -4495> 2014-05-19 18:37:03.864765 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:06.922036) v2 ==== 47+0+0 (2681713562 0 0) 0x68aaa80 con 0x4cb9340 -4494> 2014-05-19 18:37:03.864821 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:06.922036) v2 ==== 47+0+0 (2681713562 0 0) 0x5bea1c0 con 0x4cb8dc0 -4493> 2014-05-19 18:37:03.864831 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.922036) v2 -- ?+0 0x5e6ca80 con 0x4cb9340 -4492> 2014-05-19 18:37:03.864880 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4491> 2014-05-19 18:37:03.864921 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.922036) v2 -- ?+0 0x8649a40 con 0x4cb8dc0 -4490> 2014-05-19 18:37:03.864963 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4489> 2014-05-19 18:37:03.891240 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:03.891068) v2 ==== 47+0+0 (4218762491 0 0) 0x8649880 con 0x4cb86e0 -4488> 2014-05-19 18:37:03.891290 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.891068) v2 -- ?+0 0x5bea1c0 con 0x4cb86e0 -4487> 2014-05-19 18:37:03.891346 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4486> 2014-05-19 18:37:03.891288 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:03.891068) v2 ==== 47+0+0 (4218762491 0 0) 0x5e6d6c0 con 0x4cb8420 -4485> 2014-05-19 18:37:03.891376 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:03.891068) v2 -- ?+0 0x68aaa80 con 0x4cb8420 -4484> 2014-05-19 18:37:03.891413 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4483> 2014-05-19 18:37:04.494873 7fd04f907700 5 osd.0 21 tick -4482> 2014-05-19 18:37:04.494895 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4481> 2014-05-19 18:37:04.494899 7fd04f907700 10 osd.0 21 do_waiters -- start -4480> 2014-05-19 18:37:04.494902 7fd04f907700 10 osd.0 21 do_waiters -- finish -4479> 2014-05-19 18:37:04.494942 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4478> 2014-05-19 18:37:04.726982 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:07.784242) v2 ==== 47+0+0 (3670123961 0 0) 0x3438a80 con 0x61ae420 -4477> 2014-05-19 18:37:04.726991 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:07.784242) v2 ==== 47+0+0 (3670123961 0 0) 0x5497a40 con 0x618fe40 -4476> 2014-05-19 18:37:04.727021 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:07.784242) v2 -- ?+0 0x8649880 con 0x61ae420 -4475> 2014-05-19 18:37:04.727051 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4474> 2014-05-19 18:37:04.727107 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:07.784242) v2 -- ?+0 0x5e6d6c0 con 0x618fe40 -4473> 2014-05-19 18:37:04.727160 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4472> 2014-05-19 18:37:04.775751 7fd04e905700 20 KeyValueStore::op_tp worker waiting -4471> 2014-05-19 18:37:04.775774 7fd043ebd700 20 OSD::op_tp worker waiting -4470> 2014-05-19 18:37:04.876953 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:04.876772) v2 ==== 47+0+0 (3420267729 0 0) 0x64afc00 con 0x4cb82c0 -4469> 2014-05-19 18:37:04.876998 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:04.876772) v2 -- ?+0 0x5497a40 con 0x4cb82c0 -4468> 2014-05-19 18:37:04.877029 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4467> 2014-05-19 18:37:04.877114 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:04.876772) v2 ==== 47+0+0 (3420267729 0 0) 0x77e96c0 con 0x4cb8160 -4466> 2014-05-19 18:37:04.877154 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:04.876772) v2 -- ?+0 0x3438a80 con 0x4cb8160 -4465> 2014-05-19 18:37:04.877188 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4464> 2014-05-19 18:37:04.948015 7fd0396a8700 20 OSD::command_tp worker waiting -4463> 2014-05-19 18:37:04.965315 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:08.022580) v2 ==== 47+0+0 (1073992579 0 0) 0x60e2000 con 0x4cb8dc0 -4462> 2014-05-19 18:37:04.965316 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:08.022580) v2 ==== 47+0+0 (1073992579 0 0) 0x6113500 con 0x4cb9340 -4461> 2014-05-19 18:37:04.965355 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.022580) v2 -- ?+0 0x77e96c0 con 0x4cb8dc0 -4460> 2014-05-19 18:37:04.965424 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4459> 2014-05-19 18:37:04.965548 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.022580) v2 -- ?+0 0x64afc00 con 0x4cb9340 -4458> 2014-05-19 18:37:04.965595 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4457> 2014-05-19 18:37:05.227598 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:08.284893) v2 ==== 47+0+0 (4245004500 0 0) 0x64a7a40 con 0x61ae420 -4456> 2014-05-19 18:37:05.227640 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.284893) v2 -- ?+0 0x60e2000 con 0x61ae420 -4455> 2014-05-19 18:37:05.227675 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4454> 2014-05-19 18:37:05.227661 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:08.284893) v2 ==== 47+0+0 (4245004500 0 0) 0x6112e00 con 0x618fe40 -4453> 2014-05-19 18:37:05.227739 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.284893) v2 -- ?+0 0x6113500 con 0x618fe40 -4452> 2014-05-19 18:37:05.227769 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4451> 2014-05-19 18:37:05.290635 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:10.378098) v2 ==== 47+0+0 (3764765643 0 0) 0x7746540 con 0x5b21600 -4450> 2014-05-19 18:37:05.290643 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:10.378098) v2 ==== 47+0+0 (3764765643 0 0) 0x79e76c0 con 0x5f5b760 -4449> 2014-05-19 18:37:05.290687 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:10.378098) v2 -- ?+0 0x64a7a40 con 0x5b21600 -4448> 2014-05-19 18:37:05.290724 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4447> 2014-05-19 18:37:05.290780 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:10.378098) v2 -- ?+0 0x6112e00 con 0x5f5b760 -4446> 2014-05-19 18:37:05.290810 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4445> 2014-05-19 18:37:05.342362 7fd03d6b0700 20 OSD::op_tp worker waiting -4444> 2014-05-19 18:37:05.342383 7fd041eb9700 20 OSD::op_tp worker waiting -4443> 2014-05-19 18:37:05.342410 7fd03e6b2700 20 OSD::op_tp worker waiting -4442> 2014-05-19 18:37:05.342426 7fd03c6ae700 20 OSD::op_tp worker waiting -4441> 2014-05-19 18:37:05.342497 7fd0436bc700 20 OSD::op_tp worker waiting -4440> 2014-05-19 18:37:05.342612 7fd03bead700 20 OSD::op_tp worker waiting -4439> 2014-05-19 18:37:05.342833 7fd03f6b4700 20 OSD::op_tp worker waiting -4438> 2014-05-19 18:37:05.357089 7fd0446be700 20 OSD::op_tp worker waiting -4437> 2014-05-19 18:37:05.357114 7fd03feb5700 20 OSD::op_tp worker waiting -4436> 2014-05-19 18:37:05.363959 7fd040eb7700 20 OSD::op_tp worker waiting -4435> 2014-05-19 18:37:05.369428 7fd0406b6700 20 OSD::op_tp worker waiting -4434> 2014-05-19 18:37:05.377507 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:05.377311) v2 ==== 47+0+0 (3565944884 0 0) 0x77e8c40 con 0x4cb8160 -4433> 2014-05-19 18:37:05.377534 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:05.377311) v2 ==== 47+0+0 (3565944884 0 0) 0x64ae000 con 0x4cb82c0 -4432> 2014-05-19 18:37:05.377559 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:05.377311) v2 -- ?+0 0x7746540 con 0x4cb8160 -4431> 2014-05-19 18:37:05.377585 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4430> 2014-05-19 18:37:05.377637 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:05.377311) v2 -- ?+0 0x79e76c0 con 0x4cb82c0 -4429> 2014-05-19 18:37:05.377681 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4428> 2014-05-19 18:37:05.390935 7fd039ea9700 20 OSD::disk_tp worker waiting -4427> 2014-05-19 18:37:05.394488 7fd03aeab700 20 OSD::op_tp worker waiting -4426> 2014-05-19 18:37:05.394532 7fd03eeb3700 20 OSD::op_tp worker waiting -4425> 2014-05-19 18:37:05.412477 7fd042ebb700 20 OSD::op_tp worker waiting -4424> 2014-05-19 18:37:05.412499 7fd03deb1700 20 OSD::op_tp worker waiting -4423> 2014-05-19 18:37:05.421133 7fd04e104700 20 KeyValueStore::op_tp worker waiting -4422> 2014-05-19 18:37:05.421154 7fd03ceaf700 20 OSD::op_tp worker waiting -4421> 2014-05-19 18:37:05.421178 7fd03b6ac700 20 OSD::op_tp worker waiting -4420> 2014-05-19 18:37:05.421241 7fd0426ba700 20 OSD::op_tp worker waiting -4419> 2014-05-19 18:37:05.421283 7fd0416b8700 20 OSD::op_tp worker waiting -4418> 2014-05-19 18:37:05.494988 7fd04f907700 5 osd.0 21 tick -4417> 2014-05-19 18:37:05.495009 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4416> 2014-05-19 18:37:05.495014 7fd04f907700 10 osd.0 21 do_waiters -- start -4415> 2014-05-19 18:37:05.495016 7fd04f907700 10 osd.0 21 do_waiters -- finish -4414> 2014-05-19 18:37:05.495039 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4413> 2014-05-19 18:37:05.795525 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:08.852800) v2 ==== 47+0+0 (2419986421 0 0) 0x72e4fc0 con 0x61ae160 -4412> 2014-05-19 18:37:05.795529 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:08.852800) v2 ==== 47+0+0 (2419986421 0 0) 0x70ea540 con 0x61b0420 -4411> 2014-05-19 18:37:05.795569 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.852800) v2 -- ?+0 0x77e8c40 con 0x61ae160 -4410> 2014-05-19 18:37:05.795602 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4409> 2014-05-19 18:37:05.795625 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.852800) v2 -- ?+0 0x64ae000 con 0x61b0420 -4408> 2014-05-19 18:37:05.795660 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4407> 2014-05-19 18:37:06.215328 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:11.302771) v2 ==== 47+0+0 (1434277499 0 0) 0x66f8540 con 0x30946e0 -4406> 2014-05-19 18:37:06.215337 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 74 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:11.302771) v2 ==== 47+0+0 (1434277499 0 0) 0x77e8000 con 0x3094160 -4405> 2014-05-19 18:37:06.215396 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:11.302771) v2 -- ?+0 0x72e4fc0 con 0x30946e0 -4404> 2014-05-19 18:37:06.215475 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4403> 2014-05-19 18:37:06.215529 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:11.302771) v2 -- ?+0 0x70ea540 con 0x3094160 -4402> 2014-05-19 18:37:06.215556 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4401> 2014-05-19 18:37:06.495103 7fd04f907700 5 osd.0 21 tick -4400> 2014-05-19 18:37:06.495124 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4399> 2014-05-19 18:37:06.495129 7fd04f907700 10 osd.0 21 do_waiters -- start -4398> 2014-05-19 18:37:06.495131 7fd04f907700 10 osd.0 21 do_waiters -- finish -4397> 2014-05-19 18:37:06.495172 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4396> 2014-05-19 18:37:06.675780 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:11.763261) v2 ==== 47+0+0 (1709244652 0 0) 0x784ca80 con 0x31e1600 -4395> 2014-05-19 18:37:06.675790 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:11.763261) v2 ==== 47+0+0 (1709244652 0 0) 0x653ea80 con 0x61b0160 -4394> 2014-05-19 18:37:06.675829 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:11.763261) v2 -- ?+0 0x66f8540 con 0x31e1600 -4393> 2014-05-19 18:37:06.675861 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4392> 2014-05-19 18:37:06.675900 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:11.763261) v2 -- ?+0 0x77e8000 con 0x61b0160 -4391> 2014-05-19 18:37:06.675955 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4390> 2014-05-19 18:37:06.775830 7fd04e905700 20 KeyValueStore::op_tp worker waiting -4389> 2014-05-19 18:37:06.775852 7fd043ebd700 20 OSD::op_tp worker waiting -4388> 2014-05-19 18:37:06.791819 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:06.791601) v2 ==== 47+0+0 (3721296548 0 0) 0x5e6d340 con 0x4cb8420 -4387> 2014-05-19 18:37:06.791831 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:06.791601) v2 ==== 47+0+0 (3721296548 0 0) 0x8648380 con 0x4cb86e0 -4386> 2014-05-19 18:37:06.791865 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.791601) v2 -- ?+0 0x653ea80 con 0x4cb8420 -4385> 2014-05-19 18:37:06.791956 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4384> 2014-05-19 18:37:06.792039 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.791601) v2 -- ?+0 0x784ca80 con 0x4cb86e0 -4383> 2014-05-19 18:37:06.792088 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4382> 2014-05-19 18:37:06.896141 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:09.953432) v2 ==== 47+0+0 (4217277717 0 0) 0x55c88c0 con 0x61b0420 -4381> 2014-05-19 18:37:06.896156 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:09.953432) v2 ==== 47+0+0 (4217277717 0 0) 0x72e41c0 con 0x61ae160 -4380> 2014-05-19 18:37:06.896186 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:09.953432) v2 -- ?+0 0x5e6d340 con 0x61b0420 -4379> 2014-05-19 18:37:06.896222 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4378> 2014-05-19 18:37:06.896247 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:09.953432) v2 -- ?+0 0x8648380 con 0x61ae160 -4377> 2014-05-19 18:37:06.896273 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4376> 2014-05-19 18:37:06.948095 7fd0396a8700 20 OSD::command_tp worker waiting -4375> 2014-05-19 18:37:06.962411 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -4374> 2014-05-19 18:37:06.962430 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -4373> 2014-05-19 18:37:06.962438 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x7980380 con 0x3622580 -4372> 2014-05-19 18:37:06.962464 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x860f6c0 con 0x3622840 -4371> 2014-05-19 18:37:06.962501 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x73c21c0 con 0x3622b00 -4370> 2014-05-19 18:37:06.962515 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x7746c40 con 0x3622dc0 -4369> 2014-05-19 18:37:06.962528 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x692d180 con 0x30594a0 -4368> 2014-05-19 18:37:06.962545 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x6c4d6c0 con 0x3622f20 -4367> 2014-05-19 18:37:06.962559 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x7930700 con 0x5e7ef20 -4366> 2014-05-19 18:37:06.962573 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x6710fc0 con 0x5e7f1e0 -4365> 2014-05-19 18:37:06.962596 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x5b28380 con 0x5e7e2c0 -4364> 2014-05-19 18:37:06.962628 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x5e288c0 con 0x5d7bb80 -4363> 2014-05-19 18:37:06.962656 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x72e5c00 con 0x5e7ec60 -4362> 2014-05-19 18:37:06.962683 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x76adc00 con 0x531b760 -4361> 2014-05-19 18:37:06.962716 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x7890700 con 0x53bba20 -4360> 2014-05-19 18:37:06.962743 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x5666540 con 0x51cac60 -4359> 2014-05-19 18:37:06.962775 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x669ee00 con 0x5bd66e0 -4358> 2014-05-19 18:37:06.962801 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:06.962436) v2 -- ?+0 0x5dff880 con 0x5bd6840 -4357> 2014-05-19 18:37:06.962823 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x5cbcfc0 con 0x3622580 -4356> 2014-05-19 18:37:06.962938 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4355> 2014-05-19 18:37:06.962956 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x725d880 con 0x3622b00 -4354> 2014-05-19 18:37:06.963011 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4353> 2014-05-19 18:37:06.963045 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x7a5ea80 con 0x30594a0 -4352> 2014-05-19 18:37:06.963083 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4351> 2014-05-19 18:37:06.963095 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x7733c00 con 0x3622f20 -4350> 2014-05-19 18:37:06.963138 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4349> 2014-05-19 18:37:06.963152 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x60a0700 con 0x3622840 -4348> 2014-05-19 18:37:06.963207 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4347> 2014-05-19 18:37:06.963229 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x772d180 con 0x5e7ef20 -4346> 2014-05-19 18:37:06.963260 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4345> 2014-05-19 18:37:06.963267 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x6c4a700 con 0x5e7f1e0 -4344> 2014-05-19 18:37:06.963283 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4343> 2014-05-19 18:37:06.963288 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x59bcc40 con 0x5e7e2c0 -4342> 2014-05-19 18:37:06.963371 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4341> 2014-05-19 18:37:06.963388 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x5f0b180 con 0x5d7bb80 -4340> 2014-05-19 18:37:06.963414 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4339> 2014-05-19 18:37:06.963423 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x662ba40 con 0x5e7ec60 -4338> 2014-05-19 18:37:06.963439 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4337> 2014-05-19 18:37:06.963451 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x6009dc0 con 0x531b760 -4336> 2014-05-19 18:37:06.963467 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4335> 2014-05-19 18:37:06.963473 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x66f88c0 con 0x53bba20 -4334> 2014-05-19 18:37:06.963500 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4333> 2014-05-19 18:37:06.963512 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x60f68c0 con 0x51cac60 -4332> 2014-05-19 18:37:06.963530 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4331> 2014-05-19 18:37:06.963538 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x7aa4380 con 0x5bd66e0 -4330> 2014-05-19 18:37:06.963552 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4329> 2014-05-19 18:37:06.963561 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 81 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x656cc40 con 0x5bd6840 -4328> 2014-05-19 18:37:06.963575 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4327> 2014-05-19 18:37:06.963581 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:06.962436) v2 ==== 47+0+0 (1134888952 0 0) 0x5f7a8c0 con 0x3622dc0 -4326> 2014-05-19 18:37:06.963598 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4325> 2014-05-19 18:37:07.292285 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:07.292122) v2 ==== 47+0+0 (3921546508 0 0) 0x70eb340 con 0x4cb86e0 -4324> 2014-05-19 18:37:07.292373 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:07.292122) v2 -- ?+0 0x72e41c0 con 0x4cb86e0 -4323> 2014-05-19 18:37:07.292320 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:07.292122) v2 ==== 47+0+0 (3921546508 0 0) 0x5e6c540 con 0x4cb8420 -4322> 2014-05-19 18:37:07.292412 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4321> 2014-05-19 18:37:07.292449 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:07.292122) v2 -- ?+0 0x55c88c0 con 0x4cb8420 -4320> 2014-05-19 18:37:07.292485 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4319> 2014-05-19 18:37:07.342459 7fd041eb9700 20 OSD::op_tp worker waiting -4318> 2014-05-19 18:37:07.342479 7fd03c6ae700 20 OSD::op_tp worker waiting -4317> 2014-05-19 18:37:07.342498 7fd03d6b0700 20 OSD::op_tp worker waiting -4316> 2014-05-19 18:37:07.342525 7fd03e6b2700 20 OSD::op_tp worker waiting -4315> 2014-05-19 18:37:07.342562 7fd0436bc700 20 OSD::op_tp worker waiting -4314> 2014-05-19 18:37:07.342677 7fd03bead700 20 OSD::op_tp worker waiting -4313> 2014-05-19 18:37:07.342901 7fd03f6b4700 20 OSD::op_tp worker waiting -4312> 2014-05-19 18:37:07.357190 7fd03feb5700 20 OSD::op_tp worker waiting -4311> 2014-05-19 18:37:07.357214 7fd0446be700 20 OSD::op_tp worker waiting -4310> 2014-05-19 18:37:07.364042 7fd040eb7700 20 OSD::op_tp worker waiting -4309> 2014-05-19 18:37:07.369526 7fd0406b6700 20 OSD::op_tp worker waiting -4308> 2014-05-19 18:37:07.391031 7fd039ea9700 20 OSD::disk_tp worker waiting -4307> 2014-05-19 18:37:07.394583 7fd03aeab700 20 OSD::op_tp worker waiting -4306> 2014-05-19 18:37:07.394613 7fd03eeb3700 20 OSD::op_tp worker waiting -4305> 2014-05-19 18:37:07.412563 7fd042ebb700 20 OSD::op_tp worker waiting -4304> 2014-05-19 18:37:07.412606 7fd03deb1700 20 OSD::op_tp worker waiting -4303> 2014-05-19 18:37:07.421233 7fd04e104700 20 KeyValueStore::op_tp worker waiting -4302> 2014-05-19 18:37:07.421235 7fd03ceaf700 20 OSD::op_tp worker waiting -4301> 2014-05-19 18:37:07.421269 7fd03b6ac700 20 OSD::op_tp worker waiting -4300> 2014-05-19 18:37:07.421320 7fd0426ba700 20 OSD::op_tp worker waiting -4299> 2014-05-19 18:37:07.421353 7fd0416b8700 20 OSD::op_tp worker waiting -4298> 2014-05-19 18:37:07.495215 7fd04f907700 5 osd.0 21 tick -4297> 2014-05-19 18:37:07.495232 7fd04f907700 7 osd.0 21 do_mon_report -4296> 2014-05-19 18:37:07.495236 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -4295> 2014-05-19 18:37:07.495239 7fd04f907700 20 osd.0 21 send_pg_stats -4294> 2014-05-19 18:37:07.495247 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4293> 2014-05-19 18:37:07.495251 7fd04f907700 10 osd.0 21 do_waiters -- start -4292> 2014-05-19 18:37:07.495253 7fd04f907700 10 osd.0 21 do_waiters -- finish -4291> 2014-05-19 18:37:07.495282 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4290> 2014-05-19 18:37:07.776163 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:12.863723) v2 ==== 47+0+0 (3001425901 0 0) 0x7122a80 con 0x61b0160 -4289> 2014-05-19 18:37:07.776215 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:12.863723) v2 -- ?+0 0x5e6c540 con 0x61b0160 -4288> 2014-05-19 18:37:07.776174 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:12.863723) v2 ==== 47+0+0 (3001425901 0 0) 0x5f308c0 con 0x31e1600 -4287> 2014-05-19 18:37:07.776319 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4286> 2014-05-19 18:37:07.776362 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:12.863723) v2 -- ?+0 0x70eb340 con 0x31e1600 -4285> 2014-05-19 18:37:07.776394 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4284> 2014-05-19 18:37:07.865796 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:10.923117) v2 ==== 47+0+0 (336877681 0 0) 0x6113180 con 0x4cb9340 -4283> 2014-05-19 18:37:07.865826 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:10.923117) v2 ==== 47+0+0 (336877681 0 0) 0x60e2a80 con 0x4cb8dc0 -4282> 2014-05-19 18:37:07.865874 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:10.923117) v2 -- ?+0 0x7122a80 con 0x4cb9340 -4281> 2014-05-19 18:37:07.865910 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4280> 2014-05-19 18:37:07.865933 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:10.923117) v2 -- ?+0 0x5f308c0 con 0x4cb8dc0 -4279> 2014-05-19 18:37:07.865964 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4278> 2014-05-19 18:37:08.191037 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:13.278587) v2 ==== 47+0+0 (987012668 0 0) 0x798e1c0 con 0x5b21600 -4277> 2014-05-19 18:37:08.191054 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:13.278587) v2 ==== 47+0+0 (987012668 0 0) 0x79e6fc0 con 0x5f5b760 -4276> 2014-05-19 18:37:08.191086 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.278587) v2 -- ?+0 0x60e2a80 con 0x5b21600 -4275> 2014-05-19 18:37:08.191134 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4274> 2014-05-19 18:37:08.191196 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.278587) v2 -- ?+0 0x6113180 con 0x5f5b760 -4273> 2014-05-19 18:37:08.191235 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4272> 2014-05-19 18:37:08.366382 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:11.423709) v2 ==== 47+0+0 (962194457 0 0) 0x5cbca80 con 0x4cb9340 -4271> 2014-05-19 18:37:08.366407 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:11.423709) v2 ==== 47+0+0 (962194457 0 0) 0x36f7880 con 0x4cb8dc0 -4270> 2014-05-19 18:37:08.366433 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:11.423709) v2 -- ?+0 0x79e6fc0 con 0x4cb9340 -4269> 2014-05-19 18:37:08.366481 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4268> 2014-05-19 18:37:08.366504 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:11.423709) v2 -- ?+0 0x798e1c0 con 0x4cb8dc0 -4267> 2014-05-19 18:37:08.366547 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4266> 2014-05-19 18:37:08.495338 7fd04f907700 5 osd.0 21 tick -4265> 2014-05-19 18:37:08.495358 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4264> 2014-05-19 18:37:08.495363 7fd04f907700 10 osd.0 21 do_waiters -- start -4263> 2014-05-19 18:37:08.495365 7fd04f907700 10 osd.0 21 do_waiters -- finish -4262> 2014-05-19 18:37:08.495406 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4261> 2014-05-19 18:37:08.662948 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -4260> 2014-05-19 18:37:08.662966 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -4259> 2014-05-19 18:37:08.662974 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x60088c0 con 0x3622580 -4258> 2014-05-19 18:37:08.663000 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x6b71dc0 con 0x3622840 -4257> 2014-05-19 18:37:08.663037 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x51a6a80 con 0x3622b00 -4256> 2014-05-19 18:37:08.663053 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x711e8c0 con 0x3622dc0 -4255> 2014-05-19 18:37:08.663067 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x6c4c000 con 0x30594a0 -4254> 2014-05-19 18:37:08.663082 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x59bcfc0 con 0x3622f20 -4253> 2014-05-19 18:37:08.663099 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x6fe7a40 con 0x5e7ef20 -4252> 2014-05-19 18:37:08.663125 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x5e28c40 con 0x5e7f1e0 -4251> 2014-05-19 18:37:08.663150 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x725da40 con 0x5e7e2c0 -4250> 2014-05-19 18:37:08.663174 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x60a1a40 con 0x5d7bb80 -4249> 2014-05-19 18:37:08.663199 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x656ca80 con 0x5e7ec60 -4248> 2014-05-19 18:37:08.663225 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x7aa4fc0 con 0x531b760 -4247> 2014-05-19 18:37:08.663250 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x6542c40 con 0x53bba20 -4246> 2014-05-19 18:37:08.663275 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x66f8380 con 0x51cac60 -4245> 2014-05-19 18:37:08.663302 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x6009a40 con 0x5bd66e0 -4244> 2014-05-19 18:37:08.663326 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:08.662973) v2 -- ?+0 0x662b180 con 0x5bd6840 -4243> 2014-05-19 18:37:08.663377 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x725ca80 con 0x3622b00 -4242> 2014-05-19 18:37:08.663424 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4241> 2014-05-19 18:37:08.663439 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x5cbc000 con 0x3622580 -4240> 2014-05-19 18:37:08.663473 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4239> 2014-05-19 18:37:08.663489 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x60a0e00 con 0x3622840 -4238> 2014-05-19 18:37:08.663508 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4237> 2014-05-19 18:37:08.663522 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x5f7b340 con 0x3622dc0 -4236> 2014-05-19 18:37:08.663554 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4235> 2014-05-19 18:37:08.663652 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x8389880 con 0x5e7f1e0 -4234> 2014-05-19 18:37:08.663703 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4233> 2014-05-19 18:37:08.663723 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x5f0ba40 con 0x5d7bb80 -4232> 2014-05-19 18:37:08.663750 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4231> 2014-05-19 18:37:08.663770 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x798fdc0 con 0x5e7ec60 -4230> 2014-05-19 18:37:08.663793 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4229> 2014-05-19 18:37:08.663801 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x6008e00 con 0x531b760 -4228> 2014-05-19 18:37:08.663818 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4227> 2014-05-19 18:37:08.663826 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x66f8e00 con 0x53bba20 -4226> 2014-05-19 18:37:08.663841 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4225> 2014-05-19 18:37:08.663848 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x60f61c0 con 0x51cac60 -4224> 2014-05-19 18:37:08.663865 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4223> 2014-05-19 18:37:08.663874 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x7735340 con 0x5bd66e0 -4222> 2014-05-19 18:37:08.663891 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4221> 2014-05-19 18:37:08.663899 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 82 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x656c000 con 0x5bd6840 -4220> 2014-05-19 18:37:08.663916 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4219> 2014-05-19 18:37:08.663924 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x7a5e000 con 0x30594a0 -4218> 2014-05-19 18:37:08.663941 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4217> 2014-05-19 18:37:08.663949 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x7733dc0 con 0x3622f20 -4216> 2014-05-19 18:37:08.663966 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4215> 2014-05-19 18:37:08.663974 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x772d340 con 0x5e7ef20 -4214> 2014-05-19 18:37:08.663991 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4213> 2014-05-19 18:37:08.664001 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.662973) v2 ==== 47+0+0 (2661684022 0 0) 0x59bddc0 con 0x5e7e2c0 -4212> 2014-05-19 18:37:08.664018 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4211> 2014-05-19 18:37:08.728155 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:11.785464) v2 ==== 47+0+0 (20948694 0 0) 0x71cafc0 con 0x61ae420 -4210> 2014-05-19 18:37:08.728164 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:11.785464) v2 ==== 47+0+0 (20948694 0 0) 0x8388000 con 0x618fe40 -4209> 2014-05-19 18:37:08.728219 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:11.785464) v2 -- ?+0 0x36f7880 con 0x61ae420 -4208> 2014-05-19 18:37:08.728249 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4207> 2014-05-19 18:37:08.728301 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:11.785464) v2 -- ?+0 0x5cbca80 con 0x618fe40 -4206> 2014-05-19 18:37:08.728333 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4205> 2014-05-19 18:37:08.775908 7fd04e905700 20 KeyValueStore::op_tp worker waiting -4204> 2014-05-19 18:37:08.775931 7fd043ebd700 20 OSD::op_tp worker waiting -4203> 2014-05-19 18:37:08.878061 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:08.877873) v2 ==== 47+0+0 (1540207691 0 0) 0x64aec40 con 0x4cb82c0 -4202> 2014-05-19 18:37:08.878103 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.877873) v2 -- ?+0 0x8388000 con 0x4cb82c0 -4201> 2014-05-19 18:37:08.878115 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:08.877873) v2 ==== 47+0+0 (1540207691 0 0) 0x77e81c0 con 0x4cb8160 -4200> 2014-05-19 18:37:08.878184 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4199> 2014-05-19 18:37:08.878211 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:08.877873) v2 -- ?+0 0x71cafc0 con 0x4cb8160 -4198> 2014-05-19 18:37:08.878333 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4197> 2014-05-19 18:37:08.948174 7fd0396a8700 20 OSD::command_tp worker waiting -4196> 2014-05-19 18:37:09.115725 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:14.203263) v2 ==== 47+0+0 (280985151 0 0) 0x66f9c00 con 0x30946e0 -4195> 2014-05-19 18:37:09.115753 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 75 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:14.203263) v2 ==== 47+0+0 (280985151 0 0) 0x5c468c0 con 0x3094160 -4194> 2014-05-19 18:37:09.115779 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:14.203263) v2 -- ?+0 0x77e81c0 con 0x30946e0 -4193> 2014-05-19 18:37:09.115807 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4192> 2014-05-19 18:37:09.115831 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:14.203263) v2 -- ?+0 0x64aec40 con 0x3094160 -4191> 2014-05-19 18:37:09.115866 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4190> 2014-05-19 18:37:09.342558 7fd041eb9700 20 OSD::op_tp worker waiting -4189> 2014-05-19 18:37:09.342579 7fd03c6ae700 20 OSD::op_tp worker waiting -4188> 2014-05-19 18:37:09.342594 7fd03d6b0700 20 OSD::op_tp worker waiting -4187> 2014-05-19 18:37:09.342617 7fd03e6b2700 20 OSD::op_tp worker waiting -4186> 2014-05-19 18:37:09.342628 7fd0436bc700 20 OSD::op_tp worker waiting -4185> 2014-05-19 18:37:09.342740 7fd03bead700 20 OSD::op_tp worker waiting -4184> 2014-05-19 18:37:09.342966 7fd03f6b4700 20 OSD::op_tp worker waiting -4183> 2014-05-19 18:37:09.357292 7fd03feb5700 20 OSD::op_tp worker waiting -4182> 2014-05-19 18:37:09.357334 7fd0446be700 20 OSD::op_tp worker waiting -4181> 2014-05-19 18:37:09.364126 7fd040eb7700 20 OSD::op_tp worker waiting -4180> 2014-05-19 18:37:09.369623 7fd0406b6700 20 OSD::op_tp worker waiting -4179> 2014-05-19 18:37:09.391127 7fd039ea9700 20 OSD::disk_tp worker waiting -4178> 2014-05-19 18:37:09.394665 7fd03aeab700 20 OSD::op_tp worker waiting -4177> 2014-05-19 18:37:09.394693 7fd03eeb3700 20 OSD::op_tp worker waiting -4176> 2014-05-19 18:37:09.412681 7fd042ebb700 20 OSD::op_tp worker waiting -4175> 2014-05-19 18:37:09.412707 7fd03deb1700 20 OSD::op_tp worker waiting -4174> 2014-05-19 18:37:09.421345 7fd04e104700 20 KeyValueStore::op_tp worker waiting -4173> 2014-05-19 18:37:09.421348 7fd03ceaf700 20 OSD::op_tp worker waiting -4172> 2014-05-19 18:37:09.421403 7fd0426ba700 20 OSD::op_tp worker waiting -4171> 2014-05-19 18:37:09.421419 7fd03b6ac700 20 OSD::op_tp worker waiting -4170> 2014-05-19 18:37:09.421446 7fd0416b8700 20 OSD::op_tp worker waiting -4169> 2014-05-19 18:37:09.495450 7fd04f907700 5 osd.0 21 tick -4168> 2014-05-19 18:37:09.495471 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4167> 2014-05-19 18:37:09.495476 7fd04f907700 10 osd.0 21 do_waiters -- start -4166> 2014-05-19 18:37:09.495478 7fd04f907700 10 osd.0 21 do_waiters -- finish -4165> 2014-05-19 18:37:09.495519 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4164> 2014-05-19 18:37:09.796760 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:12.854091) v2 ==== 47+0+0 (3164214570 0 0) 0x55c9500 con 0x61b0420 -4163> 2014-05-19 18:37:09.796796 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:12.854091) v2 ==== 47+0+0 (3164214570 0 0) 0x73c16c0 con 0x61ae160 -4162> 2014-05-19 18:37:09.796834 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:12.854091) v2 -- ?+0 0x5c468c0 con 0x61b0420 -4161> 2014-05-19 18:37:09.796872 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4160> 2014-05-19 18:37:09.796892 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:12.854091) v2 -- ?+0 0x66f9c00 con 0x61ae160 -4159> 2014-05-19 18:37:09.796941 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4158> 2014-05-19 18:37:09.891468 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:14.979072) v2 ==== 47+0+0 (1794169418 0 0) 0x5519500 con 0x5f5b760 -4157> 2014-05-19 18:37:09.891485 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:14.979072) v2 ==== 47+0+0 (1794169418 0 0) 0x5beaa80 con 0x5b21600 -4156> 2014-05-19 18:37:09.891529 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:14.979072) v2 -- ?+0 0x55c9500 con 0x5f5b760 -4155> 2014-05-19 18:37:09.891566 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4154> 2014-05-19 18:37:09.891590 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:14.979072) v2 -- ?+0 0x73c16c0 con 0x5b21600 -4153> 2014-05-19 18:37:09.891621 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4152> 2014-05-19 18:37:09.907633 7fd0456c0700 10 monclient: tick -4151> 2014-05-19 18:37:09.907646 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:36:39.907645) -4150> 2014-05-19 18:37:09.907664 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:37:09.907664; renew after: 2014-05-19 18:38:19.906664) -- no -4149> 2014-05-19 18:37:10.297299 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:13.354620) v2 ==== 47+0+0 (1715748432 0 0) 0x55c9a40 con 0x61b0420 -4148> 2014-05-19 18:37:10.297347 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.354620) v2 -- ?+0 0x5519500 con 0x61b0420 -4147> 2014-05-19 18:37:10.297382 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4146> 2014-05-19 18:37:10.297405 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:13.354620) v2 ==== 47+0+0 (1715748432 0 0) 0x73c0540 con 0x61ae160 -4145> 2014-05-19 18:37:10.297436 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.354620) v2 -- ?+0 0x5beaa80 con 0x61ae160 -4144> 2014-05-19 18:37:10.297469 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -4143> 2014-05-19 18:37:10.428705 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:13.486020) v2 ==== 47+0+0 (3495118695 0 0) 0x788b180 con 0x618fe40 -4142> 2014-05-19 18:37:10.428743 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.486020) v2 -- ?+0 0x55c9a40 con 0x618fe40 -4141> 2014-05-19 18:37:10.428772 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4140> 2014-05-19 18:37:10.428796 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:13.486020) v2 ==== 47+0+0 (3495118695 0 0) 0x6543180 con 0x61ae420 -4139> 2014-05-19 18:37:10.428823 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.486020) v2 -- ?+0 0x73c0540 con 0x61ae420 -4138> 2014-05-19 18:37:10.428844 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -4137> 2014-05-19 18:37:10.495559 7fd04f907700 5 osd.0 21 tick -4136> 2014-05-19 18:37:10.495609 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.03 < max 0.5 = yes -4135> 2014-05-19 18:37:10.495622 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -4134> 2014-05-19 18:37:10.495628 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4133> 2014-05-19 18:37:10.495626 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 273.22 < min (86400 seconds) -4132> 2014-05-19 18:37:10.495644 7fd04f907700 20 osd.0 21 sched_scrub done -4131> 2014-05-19 18:37:10.495647 7fd04f907700 10 osd.0 21 do_waiters -- start -4130> 2014-05-19 18:37:10.495649 7fd04f907700 10 osd.0 21 do_waiters -- finish -4129> 2014-05-19 18:37:10.578634 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:10.578453) v2 ==== 47+0+0 (2114107142 0 0) 0x64ae8c0 con 0x4cb82c0 -4128> 2014-05-19 18:37:10.578640 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:10.578453) v2 ==== 47+0+0 (2114107142 0 0) 0x77e8e00 con 0x4cb8160 -4127> 2014-05-19 18:37:10.578676 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:10.578453) v2 -- ?+0 0x788b180 con 0x4cb82c0 -4126> 2014-05-19 18:37:10.578721 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4125> 2014-05-19 18:37:10.578743 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:10.578453) v2 -- ?+0 0x6543180 con 0x4cb8160 -4124> 2014-05-19 18:37:10.578798 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -4123> 2014-05-19 18:37:10.676606 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:15.764230) v2 ==== 47+0+0 (3838503566 0 0) 0x649fa40 con 0x31e1600 -4122> 2014-05-19 18:37:10.676636 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:15.764230) v2 ==== 47+0+0 (3838503566 0 0) 0x76ac1c0 con 0x61b0160 -4121> 2014-05-19 18:37:10.676654 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:15.764230) v2 -- ?+0 0x77e8e00 con 0x31e1600 -4120> 2014-05-19 18:37:10.676684 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4119> 2014-05-19 18:37:10.676701 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:15.764230) v2 -- ?+0 0x64ae8c0 con 0x61b0160 -4118> 2014-05-19 18:37:10.676731 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4117> 2014-05-19 18:37:10.775997 7fd04e905700 20 KeyValueStore::op_tp worker waiting -4116> 2014-05-19 18:37:10.776004 7fd043ebd700 20 OSD::op_tp worker waiting -4115> 2014-05-19 18:37:10.792836 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:10.792649) v2 ==== 47+0+0 (668927138 0 0) 0x5f0aa80 con 0x4cb86e0 -4114> 2014-05-19 18:37:10.792865 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:10.792649) v2 ==== 47+0+0 (668927138 0 0) 0x656c540 con 0x4cb8420 -4113> 2014-05-19 18:37:10.792899 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:10.792649) v2 -- ?+0 0x649fa40 con 0x4cb86e0 -4112> 2014-05-19 18:37:10.792938 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4111> 2014-05-19 18:37:10.793023 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:10.792649) v2 -- ?+0 0x76ac1c0 con 0x4cb8420 -4110> 2014-05-19 18:37:10.793076 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4109> 2014-05-19 18:37:10.816120 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:15.903756) v2 ==== 47+0+0 (2499608654 0 0) 0x6113dc0 con 0x30946e0 -4108> 2014-05-19 18:37:10.816150 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 76 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:15.903756) v2 ==== 47+0+0 (2499608654 0 0) 0x5c47880 con 0x3094160 -4107> 2014-05-19 18:37:10.816168 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:15.903756) v2 -- ?+0 0x5f0aa80 con 0x30946e0 -4106> 2014-05-19 18:37:10.816254 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4105> 2014-05-19 18:37:10.816316 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:15.903756) v2 -- ?+0 0x656c540 con 0x3094160 -4104> 2014-05-19 18:37:10.816373 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4103> 2014-05-19 18:37:10.948255 7fd0396a8700 20 OSD::command_tp worker waiting -4102> 2014-05-19 18:37:10.991853 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:16.079496) v2 ==== 47+0+0 (3729516539 0 0) 0x5bea1c0 con 0x5b21600 -4101> 2014-05-19 18:37:10.991911 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:16.079496) v2 -- ?+0 0x6113dc0 con 0x5b21600 -4100> 2014-05-19 18:37:10.991849 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:16.079496) v2 ==== 47+0+0 (3729516539 0 0) 0x7a36700 con 0x5f5b760 -4099> 2014-05-19 18:37:10.991995 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4098> 2014-05-19 18:37:10.992045 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:16.079496) v2 -- ?+0 0x5c47880 con 0x5f5b760 -4097> 2014-05-19 18:37:10.992077 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -4096> 2014-05-19 18:37:11.177063 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:16.264713) v2 ==== 47+0+0 (1508623621 0 0) 0x76addc0 con 0x61b0160 -4095> 2014-05-19 18:37:11.177081 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:16.264713) v2 ==== 47+0+0 (1508623621 0 0) 0x6113880 con 0x31e1600 -4094> 2014-05-19 18:37:11.177122 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:16.264713) v2 -- ?+0 0x7a36700 con 0x61b0160 -4093> 2014-05-19 18:37:11.177163 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4092> 2014-05-19 18:37:11.177189 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:16.264713) v2 -- ?+0 0x5bea1c0 con 0x31e1600 -4091> 2014-05-19 18:37:11.177232 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -4090> 2014-05-19 18:37:11.342655 7fd041eb9700 20 OSD::op_tp worker waiting -4089> 2014-05-19 18:37:11.342678 7fd0436bc700 20 OSD::op_tp worker waiting -4088> 2014-05-19 18:37:11.342729 7fd03d6b0700 20 OSD::op_tp worker waiting -4087> 2014-05-19 18:37:11.342752 7fd03c6ae700 20 OSD::op_tp worker waiting -4086> 2014-05-19 18:37:11.342804 7fd03bead700 20 OSD::op_tp worker waiting -4085> 2014-05-19 18:37:11.342820 7fd03e6b2700 20 OSD::op_tp worker waiting -4084> 2014-05-19 18:37:11.343030 7fd03f6b4700 20 OSD::op_tp worker waiting -4083> 2014-05-19 18:37:11.357389 7fd03feb5700 20 OSD::op_tp worker waiting -4082> 2014-05-19 18:37:11.357408 7fd0446be700 20 OSD::op_tp worker waiting -4081> 2014-05-19 18:37:11.364210 7fd040eb7700 20 OSD::op_tp worker waiting -4080> 2014-05-19 18:37:11.369704 7fd0406b6700 20 OSD::op_tp worker waiting -4079> 2014-05-19 18:37:11.391223 7fd039ea9700 20 OSD::disk_tp worker waiting -4078> 2014-05-19 18:37:11.394764 7fd03aeab700 20 OSD::op_tp worker waiting -4077> 2014-05-19 18:37:11.394790 7fd03eeb3700 20 OSD::op_tp worker waiting -4076> 2014-05-19 18:37:11.412782 7fd042ebb700 20 OSD::op_tp worker waiting -4075> 2014-05-19 18:37:11.412809 7fd03deb1700 20 OSD::op_tp worker waiting -4074> 2014-05-19 18:37:11.421453 7fd04e104700 20 KeyValueStore::op_tp worker waiting -4073> 2014-05-19 18:37:11.421457 7fd03ceaf700 20 OSD::op_tp worker waiting -4072> 2014-05-19 18:37:11.421497 7fd03b6ac700 20 OSD::op_tp worker waiting -4071> 2014-05-19 18:37:11.421523 7fd0416b8700 20 OSD::op_tp worker waiting -4070> 2014-05-19 18:37:11.421547 7fd0426ba700 20 OSD::op_tp worker waiting -4069> 2014-05-19 18:37:11.495719 7fd04f907700 5 osd.0 21 tick -4068> 2014-05-19 18:37:11.495740 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4067> 2014-05-19 18:37:11.495745 7fd04f907700 10 osd.0 21 do_waiters -- start -4066> 2014-05-19 18:37:11.495747 7fd04f907700 10 osd.0 21 do_waiters -- finish -4065> 2014-05-19 18:37:11.495787 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4064> 2014-05-19 18:37:11.866871 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:14.924226) v2 ==== 47+0+0 (1812494209 0 0) 0x36f7340 con 0x4cb8dc0 -4063> 2014-05-19 18:37:11.866923 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:14.924226) v2 -- ?+0 0x6113880 con 0x4cb8dc0 -4062> 2014-05-19 18:37:11.866948 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4061> 2014-05-19 18:37:11.866973 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:14.924226) v2 ==== 47+0+0 (1812494209 0 0) 0x5cbc380 con 0x4cb9340 -4060> 2014-05-19 18:37:11.867005 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:14.924226) v2 -- ?+0 0x76addc0 con 0x4cb9340 -4059> 2014-05-19 18:37:11.867028 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4058> 2014-05-19 18:37:11.916649 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:17.004301) v2 ==== 47+0+0 (3812648740 0 0) 0x5debc00 con 0x30946e0 -4057> 2014-05-19 18:37:11.916689 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:17.004301) v2 -- ?+0 0x36f7340 con 0x30946e0 -4056> 2014-05-19 18:37:11.916725 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4055> 2014-05-19 18:37:11.916708 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 77 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:17.004301) v2 ==== 47+0+0 (3812648740 0 0) 0x70a9a40 con 0x3094160 -4054> 2014-05-19 18:37:11.916741 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:17.004301) v2 -- ?+0 0x5cbc380 con 0x3094160 -4053> 2014-05-19 18:37:11.916769 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -4052> 2014-05-19 18:37:12.493360 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:12.493206) v2 ==== 47+0+0 (815892077 0 0) 0x662bc00 con 0x4cb86e0 -4051> 2014-05-19 18:37:12.493407 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:12.493206) v2 -- ?+0 0x5debc00 con 0x4cb86e0 -4050> 2014-05-19 18:37:12.493408 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:12.493206) v2 ==== 47+0+0 (815892077 0 0) 0x656d880 con 0x4cb8420 -4049> 2014-05-19 18:37:12.493443 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4048> 2014-05-19 18:37:12.493486 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:12.493206) v2 -- ?+0 0x70a9a40 con 0x4cb8420 -4047> 2014-05-19 18:37:12.493522 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -4046> 2014-05-19 18:37:12.495784 7fd04f907700 5 osd.0 21 tick -4045> 2014-05-19 18:37:12.495797 7fd04f907700 7 osd.0 21 do_mon_report -4044> 2014-05-19 18:37:12.495800 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -4043> 2014-05-19 18:37:12.495803 7fd04f907700 20 osd.0 21 send_pg_stats -4042> 2014-05-19 18:37:12.495809 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -4041> 2014-05-19 18:37:12.495812 7fd04f907700 10 osd.0 21 do_waiters -- start -4040> 2014-05-19 18:37:12.495814 7fd04f907700 10 osd.0 21 do_waiters -- finish -4039> 2014-05-19 18:37:12.495849 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4038> 2014-05-19 18:37:12.776088 7fd04e905700 20 KeyValueStore::op_tp worker waiting -4037> 2014-05-19 18:37:12.776092 7fd043ebd700 20 OSD::op_tp worker waiting -4036> 2014-05-19 18:37:12.948339 7fd0396a8700 20 OSD::command_tp worker waiting -4035> 2014-05-19 18:37:13.342746 7fd041eb9700 20 OSD::op_tp worker waiting -4034> 2014-05-19 18:37:13.342771 7fd0436bc700 20 OSD::op_tp worker waiting -4033> 2014-05-19 18:37:13.342801 7fd03d6b0700 20 OSD::op_tp worker waiting -4032> 2014-05-19 18:37:13.342822 7fd03c6ae700 20 OSD::op_tp worker waiting -4031> 2014-05-19 18:37:13.342867 7fd03e6b2700 20 OSD::op_tp worker waiting -4030> 2014-05-19 18:37:13.342896 7fd03bead700 20 OSD::op_tp worker waiting -4029> 2014-05-19 18:37:13.343105 7fd03f6b4700 20 OSD::op_tp worker waiting -4028> 2014-05-19 18:37:13.357496 7fd03feb5700 20 OSD::op_tp worker waiting -4027> 2014-05-19 18:37:13.357522 7fd0446be700 20 OSD::op_tp worker waiting -4026> 2014-05-19 18:37:13.364293 7fd040eb7700 20 OSD::op_tp worker waiting -4025> 2014-05-19 18:37:13.369802 7fd0406b6700 20 OSD::op_tp worker waiting -4024> 2014-05-19 18:37:13.391318 7fd039ea9700 20 OSD::disk_tp worker waiting -4023> 2014-05-19 18:37:13.394851 7fd03aeab700 20 OSD::op_tp worker waiting -4022> 2014-05-19 18:37:13.394872 7fd03eeb3700 20 OSD::op_tp worker waiting -4021> 2014-05-19 18:37:13.412882 7fd042ebb700 20 OSD::op_tp worker waiting -4020> 2014-05-19 18:37:13.412906 7fd03deb1700 20 OSD::op_tp worker waiting -4019> 2014-05-19 18:37:13.421563 7fd04e104700 20 KeyValueStore::op_tp worker waiting -4018> 2014-05-19 18:37:13.421564 7fd03ceaf700 20 OSD::op_tp worker waiting -4017> 2014-05-19 18:37:13.421636 7fd0416b8700 20 OSD::op_tp worker waiting -4016> 2014-05-19 18:37:13.421659 7fd0426ba700 20 OSD::op_tp worker waiting -4015> 2014-05-19 18:37:13.421683 7fd03b6ac700 20 OSD::op_tp worker waiting -4014> 2014-05-19 18:37:13.495882 7fd04f907700 5 osd.0 21 tick -4013> 2014-05-19 18:37:13.495931 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.03 < max 0.5 = yes -4012> 2014-05-19 18:37:13.495944 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -4011> 2014-05-19 18:37:13.495951 7fd03a6aa700 20 OSD::recovery_tp worker waiting -4010> 2014-05-19 18:37:13.495948 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 276.22 < min (86400 seconds) -4009> 2014-05-19 18:37:13.495967 7fd04f907700 20 osd.0 21 sched_scrub done -4008> 2014-05-19 18:37:13.495971 7fd04f907700 10 osd.0 21 do_waiters -- start -4007> 2014-05-19 18:37:13.495972 7fd04f907700 10 osd.0 21 do_waiters -- finish -4006> 2014-05-19 18:37:13.567384 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:16.624735) v2 ==== 47+0+0 (1253831753 0 0) 0x36f68c0 con 0x4cb8dc0 -4005> 2014-05-19 18:37:13.567385 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:16.624735) v2 ==== 47+0+0 (1253831753 0 0) 0x6e188c0 con 0x4cb9340 -4004> 2014-05-19 18:37:13.567427 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:16.624735) v2 -- ?+0 0x662bc00 con 0x4cb8dc0 -4003> 2014-05-19 18:37:13.567470 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4002> 2014-05-19 18:37:13.567493 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:16.624735) v2 -- ?+0 0x656d880 con 0x4cb9340 -4001> 2014-05-19 18:37:13.567541 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -4000> 2014-05-19 18:37:13.797784 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:16.855137) v2 ==== 47+0+0 (857774519 0 0) 0x55c8700 con 0x61b0420 -3999> 2014-05-19 18:37:13.797786 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:16.855137) v2 ==== 47+0+0 (857774519 0 0) 0x5dffdc0 con 0x61ae160 -3998> 2014-05-19 18:37:13.797831 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:16.855137) v2 -- ?+0 0x6e188c0 con 0x61b0420 -3997> 2014-05-19 18:37:13.797853 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3996> 2014-05-19 18:37:13.797904 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:16.855137) v2 -- ?+0 0x36f68c0 con 0x61ae160 -3995> 2014-05-19 18:37:13.797941 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3994> 2014-05-19 18:37:13.892168 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:18.979941) v2 ==== 47+0+0 (2559438195 0 0) 0x5bea700 con 0x5b21600 -3993> 2014-05-19 18:37:13.892169 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:18.979941) v2 ==== 47+0+0 (2559438195 0 0) 0x7a37c00 con 0x5f5b760 -3992> 2014-05-19 18:37:13.892223 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:18.979941) v2 -- ?+0 0x5dffdc0 con 0x5b21600 -3991> 2014-05-19 18:37:13.892255 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3990> 2014-05-19 18:37:13.892277 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:18.979941) v2 -- ?+0 0x55c8700 con 0x5f5b760 -3989> 2014-05-19 18:37:13.892311 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3988> 2014-05-19 18:37:13.963469 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -3987> 2014-05-19 18:37:13.963488 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -3986> 2014-05-19 18:37:13.963496 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x59bc1c0 con 0x3622580 -3985> 2014-05-19 18:37:13.963522 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x6c4a000 con 0x3622840 -3984> 2014-05-19 18:37:13.963558 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x7733340 con 0x3622b00 -3983> 2014-05-19 18:37:13.963573 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x692c000 con 0x3622dc0 -3982> 2014-05-19 18:37:13.963586 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x5667340 con 0x30594a0 -3981> 2014-05-19 18:37:13.963620 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x6c4c540 con 0x3622f20 -3980> 2014-05-19 18:37:13.963634 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x67101c0 con 0x5e7ef20 -3979> 2014-05-19 18:37:13.963657 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x60a0540 con 0x5e7f1e0 -3978> 2014-05-19 18:37:13.963686 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x6fe7880 con 0x5e7e2c0 -3977> 2014-05-19 18:37:13.963718 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x5e28e00 con 0x5d7bb80 -3976> 2014-05-19 18:37:13.963749 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x7747880 con 0x5e7ec60 -3975> 2014-05-19 18:37:13.963775 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x649ea80 con 0x531b760 -3974> 2014-05-19 18:37:13.963800 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x79801c0 con 0x53bba20 -3973> 2014-05-19 18:37:13.963830 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x6542000 con 0x51cac60 -3972> 2014-05-19 18:37:13.963856 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x66f9a40 con 0x5bd66e0 -3971> 2014-05-19 18:37:13.963881 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:13.963494) v2 -- ?+0 0x6008380 con 0x5bd6840 -3970> 2014-05-19 18:37:13.963915 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x6b71c00 con 0x3622b00 -3969> 2014-05-19 18:37:13.963962 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3968> 2014-05-19 18:37:13.963977 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x649e540 con 0x3622580 -3967> 2014-05-19 18:37:13.964008 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3966> 2014-05-19 18:37:13.964025 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x60a16c0 con 0x3622840 -3965> 2014-05-19 18:37:13.964082 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3964> 2014-05-19 18:37:13.964095 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x7a5e8c0 con 0x30594a0 -3963> 2014-05-19 18:37:13.964167 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3962> 2014-05-19 18:37:13.964186 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x5f7ac40 con 0x3622dc0 -3961> 2014-05-19 18:37:13.964219 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3960> 2014-05-19 18:37:13.964237 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x772ce00 con 0x5e7ef20 -3959> 2014-05-19 18:37:13.964267 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3958> 2014-05-19 18:37:13.964277 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x8388700 con 0x5e7f1e0 -3957> 2014-05-19 18:37:13.964303 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3956> 2014-05-19 18:37:13.964313 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x59bd500 con 0x5e7e2c0 -3955> 2014-05-19 18:37:13.964339 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3954> 2014-05-19 18:37:13.964348 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x7999dc0 con 0x5d7bb80 -3953> 2014-05-19 18:37:13.964380 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3952> 2014-05-19 18:37:13.964393 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x798f880 con 0x5e7ec60 -3951> 2014-05-19 18:37:13.964421 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3950> 2014-05-19 18:37:13.964437 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x60096c0 con 0x531b760 -3949> 2014-05-19 18:37:13.964467 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3948> 2014-05-19 18:37:13.964480 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x66f9500 con 0x53bba20 -3947> 2014-05-19 18:37:13.964511 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3946> 2014-05-19 18:37:13.964527 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x60f6700 con 0x51cac60 -3945> 2014-05-19 18:37:13.964556 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3944> 2014-05-19 18:37:13.964570 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x77341c0 con 0x5bd66e0 -3943> 2014-05-19 18:37:13.964630 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3942> 2014-05-19 18:37:13.964644 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 83 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x656c700 con 0x5bd6840 -3941> 2014-05-19 18:37:13.964675 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3940> 2014-05-19 18:37:13.964688 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:13.963494) v2 ==== 47+0+0 (4014912200 0 0) 0x77321c0 con 0x3622f20 -3939> 2014-05-19 18:37:13.964718 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3938> 2014-05-19 18:37:14.392641 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:19.480416) v2 ==== 47+0+0 (1484260205 0 0) 0x691bdc0 con 0x5f5b760 -3937> 2014-05-19 18:37:14.392698 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.480416) v2 -- ?+0 0x7a37c00 con 0x5f5b760 -3936> 2014-05-19 18:37:14.392732 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3935> 2014-05-19 18:37:14.392653 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:19.480416) v2 ==== 47+0+0 (1484260205 0 0) 0x5496c40 con 0x5b21600 -3934> 2014-05-19 18:37:14.392756 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.480416) v2 -- ?+0 0x5bea700 con 0x5b21600 -3933> 2014-05-19 18:37:14.392795 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3932> 2014-05-19 18:37:14.496046 7fd04f907700 5 osd.0 21 tick -3931> 2014-05-19 18:37:14.496068 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3930> 2014-05-19 18:37:14.496072 7fd04f907700 10 osd.0 21 do_waiters -- start -3929> 2014-05-19 18:37:14.496074 7fd04f907700 10 osd.0 21 do_waiters -- finish -3928> 2014-05-19 18:37:14.496115 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3927> 2014-05-19 18:37:14.677456 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:19.765214) v2 ==== 47+0+0 (3704158952 0 0) 0x76ac000 con 0x61b0160 -3926> 2014-05-19 18:37:14.677475 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:19.765214) v2 ==== 47+0+0 (3704158952 0 0) 0x36f7500 con 0x31e1600 -3925> 2014-05-19 18:37:14.677506 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.765214) v2 -- ?+0 0x691bdc0 con 0x61b0160 -3924> 2014-05-19 18:37:14.677541 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3923> 2014-05-19 18:37:14.677567 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.765214) v2 -- ?+0 0x5496c40 con 0x31e1600 -3922> 2014-05-19 18:37:14.677598 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3921> 2014-05-19 18:37:14.776180 7fd04e905700 20 KeyValueStore::op_tp worker waiting -3920> 2014-05-19 18:37:14.776182 7fd043ebd700 20 OSD::op_tp worker waiting -3919> 2014-05-19 18:37:14.816977 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:19.904766) v2 ==== 47+0+0 (1569749469 0 0) 0x503f880 con 0x30946e0 -3918> 2014-05-19 18:37:14.817028 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.904766) v2 -- ?+0 0x36f7500 con 0x30946e0 -3917> 2014-05-19 18:37:14.817025 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 78 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:19.904766) v2 ==== 47+0+0 (1569749469 0 0) 0x70a8700 con 0x3094160 -3916> 2014-05-19 18:37:14.817072 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3915> 2014-05-19 18:37:14.817083 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.904766) v2 -- ?+0 0x76ac000 con 0x3094160 -3914> 2014-05-19 18:37:14.817116 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3913> 2014-05-19 18:37:14.948419 7fd0396a8700 20 OSD::command_tp worker waiting -3912> 2014-05-19 18:37:15.317406 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:20.405208) v2 ==== 47+0+0 (3014058873 0 0) 0x5e28fc0 con 0x30946e0 -3911> 2014-05-19 18:37:15.317457 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:20.405208) v2 -- ?+0 0x503f880 con 0x30946e0 -3910> 2014-05-19 18:37:15.317451 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 79 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:20.405208) v2 ==== 47+0+0 (3014058873 0 0) 0x3525880 con 0x3094160 -3909> 2014-05-19 18:37:15.317494 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3908> 2014-05-19 18:37:15.317540 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:20.405208) v2 -- ?+0 0x70a8700 con 0x3094160 -3907> 2014-05-19 18:37:15.317618 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3906> 2014-05-19 18:37:15.342847 7fd041eb9700 20 OSD::op_tp worker waiting -3905> 2014-05-19 18:37:15.342869 7fd0436bc700 20 OSD::op_tp worker waiting -3904> 2014-05-19 18:37:15.342891 7fd03d6b0700 20 OSD::op_tp worker waiting -3903> 2014-05-19 18:37:15.342911 7fd03c6ae700 20 OSD::op_tp worker waiting -3902> 2014-05-19 18:37:15.342937 7fd03e6b2700 20 OSD::op_tp worker waiting -3901> 2014-05-19 18:37:15.342959 7fd03bead700 20 OSD::op_tp worker waiting -3900> 2014-05-19 18:37:15.343191 7fd03f6b4700 20 OSD::op_tp worker waiting -3899> 2014-05-19 18:37:15.357589 7fd03feb5700 20 OSD::op_tp worker waiting -3898> 2014-05-19 18:37:15.357613 7fd0446be700 20 OSD::op_tp worker waiting -3897> 2014-05-19 18:37:15.364374 7fd040eb7700 20 OSD::op_tp worker waiting -3896> 2014-05-19 18:37:15.369900 7fd0406b6700 20 OSD::op_tp worker waiting -3895> 2014-05-19 18:37:15.391414 7fd039ea9700 20 OSD::disk_tp worker waiting -3894> 2014-05-19 18:37:15.394938 7fd03aeab700 20 OSD::op_tp worker waiting -3893> 2014-05-19 18:37:15.394957 7fd03eeb3700 20 OSD::op_tp worker waiting -3892> 2014-05-19 18:37:15.412983 7fd042ebb700 20 OSD::op_tp worker waiting -3891> 2014-05-19 18:37:15.413009 7fd03deb1700 20 OSD::op_tp worker waiting -3890> 2014-05-19 18:37:15.421669 7fd04e104700 20 KeyValueStore::op_tp worker waiting -3889> 2014-05-19 18:37:15.421672 7fd03ceaf700 20 OSD::op_tp worker waiting -3888> 2014-05-19 18:37:15.421725 7fd0426ba700 20 OSD::op_tp worker waiting -3887> 2014-05-19 18:37:15.421740 7fd0416b8700 20 OSD::op_tp worker waiting -3886> 2014-05-19 18:37:15.421755 7fd03b6ac700 20 OSD::op_tp worker waiting -3885> 2014-05-19 18:37:15.496158 7fd04f907700 5 osd.0 21 tick -3884> 2014-05-19 18:37:15.496206 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.03 < max 0.5 = yes -3883> 2014-05-19 18:37:15.496219 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -3882> 2014-05-19 18:37:15.496215 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3881> 2014-05-19 18:37:15.496222 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 278.22 < min (86400 seconds) -3880> 2014-05-19 18:37:15.496241 7fd04f907700 20 osd.0 21 sched_scrub done -3879> 2014-05-19 18:37:15.496243 7fd04f907700 10 osd.0 21 do_waiters -- start -3878> 2014-05-19 18:37:15.496245 7fd04f907700 10 osd.0 21 do_waiters -- finish -3877> 2014-05-19 18:37:15.498256 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:18.555637) v2 ==== 47+0+0 (1959500687 0 0) 0x70a9340 con 0x61b0420 -3876> 2014-05-19 18:37:15.498254 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:18.555637) v2 ==== 47+0+0 (1959500687 0 0) 0x5dff340 con 0x61ae160 -3875> 2014-05-19 18:37:15.498309 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:18.555637) v2 -- ?+0 0x3525880 con 0x61b0420 -3874> 2014-05-19 18:37:15.498355 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3873> 2014-05-19 18:37:15.498379 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:18.555637) v2 -- ?+0 0x5e28fc0 con 0x61ae160 -3872> 2014-05-19 18:37:15.498418 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3871> 2014-05-19 18:37:15.729117 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:18.786543) v2 ==== 47+0+0 (2812856673 0 0) 0x662b500 con 0x61ae420 -3870> 2014-05-19 18:37:15.729163 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:18.786543) v2 ==== 47+0+0 (2812856673 0 0) 0x788a540 con 0x618fe40 -3869> 2014-05-19 18:37:15.729170 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:18.786543) v2 -- ?+0 0x5dff340 con 0x61ae420 -3868> 2014-05-19 18:37:15.729216 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3867> 2014-05-19 18:37:15.729231 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:18.786543) v2 -- ?+0 0x70a9340 con 0x618fe40 -3866> 2014-05-19 18:37:15.729265 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3865> 2014-05-19 18:37:15.879185 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:15.878989) v2 ==== 47+0+0 (3239649116 0 0) 0x77e8380 con 0x4cb8160 -3864> 2014-05-19 18:37:15.879237 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:15.878989) v2 -- ?+0 0x662b500 con 0x4cb8160 -3863> 2014-05-19 18:37:15.879274 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3862> 2014-05-19 18:37:15.879185 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:15.878989) v2 ==== 47+0+0 (3239649116 0 0) 0x64afdc0 con 0x4cb82c0 -3861> 2014-05-19 18:37:15.879345 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:15.878989) v2 -- ?+0 0x788a540 con 0x4cb82c0 -3860> 2014-05-19 18:37:15.879386 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3859> 2014-05-19 18:37:16.377858 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:21.465714) v2 ==== 47+0+0 (731470865 0 0) 0x36f6700 con 0x31e1600 -3858> 2014-05-19 18:37:16.377859 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:21.465714) v2 ==== 47+0+0 (731470865 0 0) 0x60e01c0 con 0x61b0160 -3857> 2014-05-19 18:37:16.377917 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:21.465714) v2 -- ?+0 0x77e8380 con 0x31e1600 -3856> 2014-05-19 18:37:16.377955 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3855> 2014-05-19 18:37:16.378009 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:21.465714) v2 -- ?+0 0x64afdc0 con 0x61b0160 -3854> 2014-05-19 18:37:16.378039 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3853> 2014-05-19 18:37:16.496325 7fd04f907700 5 osd.0 21 tick -3852> 2014-05-19 18:37:16.496347 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3851> 2014-05-19 18:37:16.496351 7fd04f907700 10 osd.0 21 do_waiters -- start -3850> 2014-05-19 18:37:16.496354 7fd04f907700 10 osd.0 21 do_waiters -- finish -3849> 2014-05-19 18:37:16.496395 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3848> 2014-05-19 18:37:16.776281 7fd04e905700 20 KeyValueStore::op_tp worker waiting -3847> 2014-05-19 18:37:16.776282 7fd043ebd700 20 OSD::op_tp worker waiting -3846> 2014-05-19 18:37:16.948503 7fd0396a8700 20 OSD::command_tp worker waiting -3845> 2014-05-19 18:37:17.342938 7fd041eb9700 20 OSD::op_tp worker waiting -3844> 2014-05-19 18:37:17.342960 7fd0436bc700 20 OSD::op_tp worker waiting -3843> 2014-05-19 18:37:17.342980 7fd03c6ae700 20 OSD::op_tp worker waiting -3842> 2014-05-19 18:37:17.343001 7fd03e6b2700 20 OSD::op_tp worker waiting -3841> 2014-05-19 18:37:17.343038 7fd03d6b0700 20 OSD::op_tp worker waiting -3840> 2014-05-19 18:37:17.343059 7fd03bead700 20 OSD::op_tp worker waiting -3839> 2014-05-19 18:37:17.343254 7fd03f6b4700 20 OSD::op_tp worker waiting -3838> 2014-05-19 18:37:17.357644 7fd03feb5700 20 OSD::op_tp worker waiting -3837> 2014-05-19 18:37:17.357680 7fd0446be700 20 OSD::op_tp worker waiting -3836> 2014-05-19 18:37:17.364459 7fd040eb7700 20 OSD::op_tp worker waiting -3835> 2014-05-19 18:37:17.369997 7fd0406b6700 20 OSD::op_tp worker waiting -3834> 2014-05-19 18:37:17.391509 7fd039ea9700 20 OSD::disk_tp worker waiting -3833> 2014-05-19 18:37:17.395039 7fd03aeab700 20 OSD::op_tp worker waiting -3832> 2014-05-19 18:37:17.395065 7fd03eeb3700 20 OSD::op_tp worker waiting -3831> 2014-05-19 18:37:17.413082 7fd042ebb700 20 OSD::op_tp worker waiting -3830> 2014-05-19 18:37:17.413106 7fd03deb1700 20 OSD::op_tp worker waiting -3829> 2014-05-19 18:37:17.421775 7fd04e104700 20 KeyValueStore::op_tp worker waiting -3828> 2014-05-19 18:37:17.421781 7fd03ceaf700 20 OSD::op_tp worker waiting -3827> 2014-05-19 18:37:17.421814 7fd0426ba700 20 OSD::op_tp worker waiting -3826> 2014-05-19 18:37:17.421840 7fd0416b8700 20 OSD::op_tp worker waiting -3825> 2014-05-19 18:37:17.421856 7fd03b6ac700 20 OSD::op_tp worker waiting -3824> 2014-05-19 18:37:17.496438 7fd04f907700 5 osd.0 21 tick -3823> 2014-05-19 18:37:17.496454 7fd04f907700 7 osd.0 21 do_mon_report -3822> 2014-05-19 18:37:17.496458 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -3821> 2014-05-19 18:37:17.496462 7fd04f907700 20 osd.0 21 send_pg_stats -3820> 2014-05-19 18:37:17.496470 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3819> 2014-05-19 18:37:17.496474 7fd04f907700 10 osd.0 21 do_waiters -- start -3818> 2014-05-19 18:37:17.496475 7fd04f907700 10 osd.0 21 do_waiters -- finish -3817> 2014-05-19 18:37:17.496514 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3816> 2014-05-19 18:37:17.793944 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:17.793742) v2 ==== 47+0+0 (2632730943 0 0) 0x55c21c0 con 0x4cb86e0 -3815> 2014-05-19 18:37:17.793993 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:17.793742) v2 -- ?+0 0x36f6700 con 0x4cb86e0 -3814> 2014-05-19 18:37:17.794029 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3813> 2014-05-19 18:37:17.794011 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:17.793742) v2 ==== 47+0+0 (2632730943 0 0) 0x656ce00 con 0x4cb8420 -3812> 2014-05-19 18:37:17.794053 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:17.793742) v2 -- ?+0 0x60e01c0 con 0x4cb8420 -3811> 2014-05-19 18:37:17.794088 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3810> 2014-05-19 18:37:17.892947 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:22.980868) v2 ==== 47+0+0 (3175964899 0 0) 0x691b340 con 0x5f5b760 -3809> 2014-05-19 18:37:17.892963 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:22.980868) v2 ==== 47+0+0 (3175964899 0 0) 0x5c46fc0 con 0x5b21600 -3808> 2014-05-19 18:37:17.892999 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:22.980868) v2 -- ?+0 0x656ce00 con 0x5f5b760 -3807> 2014-05-19 18:37:17.893037 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3806> 2014-05-19 18:37:17.893062 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:22.980868) v2 -- ?+0 0x55c21c0 con 0x5b21600 -3805> 2014-05-19 18:37:17.893096 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3804> 2014-05-19 18:37:18.496559 7fd04f907700 5 osd.0 21 tick -3803> 2014-05-19 18:37:18.496594 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3802> 2014-05-19 18:37:18.496599 7fd04f907700 10 osd.0 21 do_waiters -- start -3801> 2014-05-19 18:37:18.496601 7fd04f907700 10 osd.0 21 do_waiters -- finish -3800> 2014-05-19 18:37:18.496628 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3799> 2014-05-19 18:37:18.776358 7fd04e905700 20 KeyValueStore::op_tp worker waiting -3798> 2014-05-19 18:37:18.776375 7fd043ebd700 20 OSD::op_tp worker waiting -3797> 2014-05-19 18:37:18.817738 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:23.905670) v2 ==== 47+0+0 (1250935077 0 0) 0x5cbc380 con 0x30946e0 -3796> 2014-05-19 18:37:18.817752 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 80 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:23.905670) v2 ==== 47+0+0 (1250935077 0 0) 0x3524380 con 0x3094160 -3795> 2014-05-19 18:37:18.817789 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:23.905670) v2 -- ?+0 0x5c46fc0 con 0x30946e0 -3794> 2014-05-19 18:37:18.817820 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3793> 2014-05-19 18:37:18.817847 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:23.905670) v2 -- ?+0 0x691b340 con 0x3094160 -3792> 2014-05-19 18:37:18.817884 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3791> 2014-05-19 18:37:18.867766 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:21.925237) v2 ==== 47+0+0 (558880025 0 0) 0x669f340 con 0x4cb9340 -3790> 2014-05-19 18:37:18.867780 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:21.925237) v2 ==== 47+0+0 (558880025 0 0) 0x79628c0 con 0x4cb8dc0 -3789> 2014-05-19 18:37:18.867815 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:21.925237) v2 -- ?+0 0x3524380 con 0x4cb9340 -3788> 2014-05-19 18:37:18.867849 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3787> 2014-05-19 18:37:18.867870 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:21.925237) v2 -- ?+0 0x5cbc380 con 0x4cb8dc0 -3786> 2014-05-19 18:37:18.867900 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3785> 2014-05-19 18:37:18.948590 7fd0396a8700 20 OSD::command_tp worker waiting -3784> 2014-05-19 18:37:19.343035 7fd0436bc700 20 OSD::op_tp worker waiting -3783> 2014-05-19 18:37:19.343058 7fd041eb9700 20 OSD::op_tp worker waiting -3782> 2014-05-19 18:37:19.343085 7fd03e6b2700 20 OSD::op_tp worker waiting -3781> 2014-05-19 18:37:19.343120 7fd03c6ae700 20 OSD::op_tp worker waiting -3780> 2014-05-19 18:37:19.343145 7fd03d6b0700 20 OSD::op_tp worker waiting -3779> 2014-05-19 18:37:19.343193 7fd03bead700 20 OSD::op_tp worker waiting -3778> 2014-05-19 18:37:19.343310 7fd03f6b4700 20 OSD::op_tp worker waiting -3777> 2014-05-19 18:37:19.357742 7fd03feb5700 20 OSD::op_tp worker waiting -3776> 2014-05-19 18:37:19.357773 7fd0446be700 20 OSD::op_tp worker waiting -3775> 2014-05-19 18:37:19.364550 7fd040eb7700 20 OSD::op_tp worker waiting -3774> 2014-05-19 18:37:19.370095 7fd0406b6700 20 OSD::op_tp worker waiting -3773> 2014-05-19 18:37:19.391589 7fd039ea9700 20 OSD::disk_tp worker waiting -3772> 2014-05-19 18:37:19.395139 7fd03aeab700 20 OSD::op_tp worker waiting -3771> 2014-05-19 18:37:19.395168 7fd03eeb3700 20 OSD::op_tp worker waiting -3770> 2014-05-19 18:37:19.413180 7fd042ebb700 20 OSD::op_tp worker waiting -3769> 2014-05-19 18:37:19.413207 7fd03deb1700 20 OSD::op_tp worker waiting -3768> 2014-05-19 18:37:19.421873 7fd04e104700 20 KeyValueStore::op_tp worker waiting -3767> 2014-05-19 18:37:19.421877 7fd03ceaf700 20 OSD::op_tp worker waiting -3766> 2014-05-19 18:37:19.421915 7fd0416b8700 20 OSD::op_tp worker waiting -3765> 2014-05-19 18:37:19.421938 7fd0426ba700 20 OSD::op_tp worker waiting -3764> 2014-05-19 18:37:19.421962 7fd03b6ac700 20 OSD::op_tp worker waiting -3763> 2014-05-19 18:37:19.496685 7fd04f907700 5 osd.0 21 tick -3762> 2014-05-19 18:37:19.496706 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3761> 2014-05-19 18:37:19.496711 7fd04f907700 10 osd.0 21 do_waiters -- start -3760> 2014-05-19 18:37:19.496713 7fd04f907700 10 osd.0 21 do_waiters -- finish -3759> 2014-05-19 18:37:19.496737 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3758> 2014-05-19 18:37:19.593359 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:24.681322) v2 ==== 47+0+0 (2361956061 0 0) 0x5c46380 con 0x5b21600 -3757> 2014-05-19 18:37:19.593403 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:24.681322) v2 ==== 47+0+0 (2361956061 0 0) 0x71cb180 con 0x5f5b760 -3756> 2014-05-19 18:37:19.593411 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:24.681322) v2 -- ?+0 0x79628c0 con 0x5b21600 -3755> 2014-05-19 18:37:19.593510 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3754> 2014-05-19 18:37:19.593536 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:24.681322) v2 -- ?+0 0x669f340 con 0x5f5b760 -3753> 2014-05-19 18:37:19.593568 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3752> 2014-05-19 18:37:19.864017 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -3751> 2014-05-19 18:37:19.864036 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -3750> 2014-05-19 18:37:19.864043 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x662b880 con 0x3622580 -3749> 2014-05-19 18:37:19.864070 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x5667c00 con 0x3622840 -3748> 2014-05-19 18:37:19.864107 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x59bc540 con 0x3622b00 -3747> 2014-05-19 18:37:19.864122 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x73c2700 con 0x3622dc0 -3746> 2014-05-19 18:37:19.864136 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x5f0ba40 con 0x30594a0 -3745> 2014-05-19 18:37:19.864164 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x8389880 con 0x3622f20 -3744> 2014-05-19 18:37:19.864178 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x5f7b340 con 0x5e7ef20 -3743> 2014-05-19 18:37:19.864201 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x60a0e00 con 0x5e7f1e0 -3742> 2014-05-19 18:37:19.864235 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x5cbc000 con 0x5e7e2c0 -3741> 2014-05-19 18:37:19.864265 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x725ca80 con 0x5d7bb80 -3740> 2014-05-19 18:37:19.864292 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x5f7a8c0 con 0x5e7ec60 -3739> 2014-05-19 18:37:19.864319 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x656cc40 con 0x531b760 -3738> 2014-05-19 18:37:19.864344 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x7aa4380 con 0x53bba20 -3737> 2014-05-19 18:37:19.864371 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x60f68c0 con 0x51cac60 -3736> 2014-05-19 18:37:19.864397 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x66f88c0 con 0x5bd66e0 -3735> 2014-05-19 18:37:19.864422 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:19.864042) v2 -- ?+0 0x6009dc0 con 0x5bd6840 -3734> 2014-05-19 18:37:19.864445 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x55f4fc0 con 0x3622580 -3733> 2014-05-19 18:37:19.864493 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3732> 2014-05-19 18:37:19.864539 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x6b71500 con 0x3622b00 -3731> 2014-05-19 18:37:19.864568 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3730> 2014-05-19 18:37:19.864608 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x60a1180 con 0x3622840 -3729> 2014-05-19 18:37:19.864645 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3728> 2014-05-19 18:37:19.864714 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x5f328c0 con 0x3622dc0 -3727> 2014-05-19 18:37:19.864768 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3726> 2014-05-19 18:37:19.864782 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x772cc40 con 0x5e7ef20 -3725> 2014-05-19 18:37:19.864801 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3724> 2014-05-19 18:37:19.864807 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x8388540 con 0x5e7f1e0 -3723> 2014-05-19 18:37:19.864821 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3722> 2014-05-19 18:37:19.864837 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x5f06540 con 0x5e7e2c0 -3721> 2014-05-19 18:37:19.864856 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3720> 2014-05-19 18:37:19.864862 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x7999880 con 0x5d7bb80 -3719> 2014-05-19 18:37:19.864877 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3718> 2014-05-19 18:37:19.864882 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x798fc00 con 0x5e7ec60 -3717> 2014-05-19 18:37:19.864897 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3716> 2014-05-19 18:37:19.864903 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x5c46700 con 0x531b760 -3715> 2014-05-19 18:37:19.864919 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3714> 2014-05-19 18:37:19.864928 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x66f8000 con 0x53bba20 -3713> 2014-05-19 18:37:19.864943 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3712> 2014-05-19 18:37:19.864951 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x60f6e00 con 0x51cac60 -3711> 2014-05-19 18:37:19.864966 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3710> 2014-05-19 18:37:19.864974 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x7734000 con 0x5bd66e0 -3709> 2014-05-19 18:37:19.864989 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3708> 2014-05-19 18:37:19.864997 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 84 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x656c1c0 con 0x5bd6840 -3707> 2014-05-19 18:37:19.865011 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3706> 2014-05-19 18:37:19.865018 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x7a5f180 con 0x30594a0 -3705> 2014-05-19 18:37:19.865033 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3704> 2014-05-19 18:37:19.865040 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:19.864042) v2 ==== 47+0+0 (2332206984 0 0) 0x682fdc0 con 0x3622f20 -3703> 2014-05-19 18:37:19.865058 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3702> 2014-05-19 18:37:19.907749 7fd0456c0700 10 monclient: tick -3701> 2014-05-19 18:37:19.907762 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:36:49.907761) -3700> 2014-05-19 18:37:19.907782 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:37:19.907782; renew after: 2014-05-19 18:38:19.906664) -- no -3699> 2014-05-19 18:37:20.496781 7fd04f907700 5 osd.0 21 tick -3698> 2014-05-19 18:37:20.496830 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.03 < max 0.5 = yes -3697> 2014-05-19 18:37:20.496843 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -3696> 2014-05-19 18:37:20.496851 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3695> 2014-05-19 18:37:20.496846 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 283.221 < min (86400 seconds) -3694> 2014-05-19 18:37:20.496865 7fd04f907700 20 osd.0 21 sched_scrub done -3693> 2014-05-19 18:37:20.496868 7fd04f907700 10 osd.0 21 do_waiters -- start -3692> 2014-05-19 18:37:20.496870 7fd04f907700 10 osd.0 21 do_waiters -- finish -3691> 2014-05-19 18:37:20.518175 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:25.606192) v2 ==== 47+0+0 (2436163238 0 0) 0x36f7340 con 0x3094160 -3690> 2014-05-19 18:37:20.518180 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 81 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:25.606192) v2 ==== 47+0+0 (2436163238 0 0) 0x656c540 con 0x30946e0 -3689> 2014-05-19 18:37:20.518225 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.606192) v2 -- ?+0 0x71cb180 con 0x3094160 -3688> 2014-05-19 18:37:20.518263 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3687> 2014-05-19 18:37:20.518317 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.606192) v2 -- ?+0 0x5c46380 con 0x30946e0 -3686> 2014-05-19 18:37:20.518353 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3685> 2014-05-19 18:37:20.776455 7fd04e905700 20 KeyValueStore::op_tp worker waiting -3684> 2014-05-19 18:37:20.776457 7fd043ebd700 20 OSD::op_tp worker waiting -3683> 2014-05-19 18:37:20.798726 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:23.856186) v2 ==== 47+0+0 (114767513 0 0) 0x70a8540 con 0x61b0420 -3682> 2014-05-19 18:37:20.798776 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:23.856186) v2 -- ?+0 0x36f7340 con 0x61b0420 -3681> 2014-05-19 18:37:20.798804 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3680> 2014-05-19 18:37:20.798734 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:23.856186) v2 ==== 47+0+0 (114767513 0 0) 0x5dfe8c0 con 0x61ae160 -3679> 2014-05-19 18:37:20.798827 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:23.856186) v2 -- ?+0 0x656c540 con 0x61ae160 -3678> 2014-05-19 18:37:20.798844 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3677> 2014-05-19 18:37:20.948680 7fd0396a8700 20 OSD::command_tp worker waiting -3676> 2014-05-19 18:37:21.343132 7fd041eb9700 20 OSD::op_tp worker waiting -3675> 2014-05-19 18:37:21.343156 7fd0436bc700 20 OSD::op_tp worker waiting -3674> 2014-05-19 18:37:21.343173 7fd03e6b2700 20 OSD::op_tp worker waiting -3673> 2014-05-19 18:37:21.343194 7fd03c6ae700 20 OSD::op_tp worker waiting -3672> 2014-05-19 18:37:21.343208 7fd03d6b0700 20 OSD::op_tp worker waiting -3671> 2014-05-19 18:37:21.343263 7fd03bead700 20 OSD::op_tp worker waiting -3670> 2014-05-19 18:37:21.343374 7fd03f6b4700 20 OSD::op_tp worker waiting -3669> 2014-05-19 18:37:21.357818 7fd03feb5700 20 OSD::op_tp worker waiting -3668> 2014-05-19 18:37:21.357846 7fd0446be700 20 OSD::op_tp worker waiting -3667> 2014-05-19 18:37:21.364646 7fd040eb7700 20 OSD::op_tp worker waiting -3666> 2014-05-19 18:37:21.370191 7fd0406b6700 20 OSD::op_tp worker waiting -3665> 2014-05-19 18:37:21.391669 7fd039ea9700 20 OSD::disk_tp worker waiting -3664> 2014-05-19 18:37:21.395239 7fd03aeab700 20 OSD::op_tp worker waiting -3663> 2014-05-19 18:37:21.395263 7fd03eeb3700 20 OSD::op_tp worker waiting -3662> 2014-05-19 18:37:21.413280 7fd042ebb700 20 OSD::op_tp worker waiting -3661> 2014-05-19 18:37:21.413304 7fd03deb1700 20 OSD::op_tp worker waiting -3660> 2014-05-19 18:37:21.421980 7fd04e104700 20 KeyValueStore::op_tp worker waiting -3659> 2014-05-19 18:37:21.421983 7fd03ceaf700 20 OSD::op_tp worker waiting -3658> 2014-05-19 18:37:21.422030 7fd03b6ac700 20 OSD::op_tp worker waiting -3657> 2014-05-19 18:37:21.422053 7fd0426ba700 20 OSD::op_tp worker waiting -3656> 2014-05-19 18:37:21.422075 7fd0416b8700 20 OSD::op_tp worker waiting -3655> 2014-05-19 18:37:21.496955 7fd04f907700 5 osd.0 21 tick -3654> 2014-05-19 18:37:21.496976 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3653> 2014-05-19 18:37:21.496980 7fd04f907700 10 osd.0 21 do_waiters -- start -3652> 2014-05-19 18:37:21.496982 7fd04f907700 10 osd.0 21 do_waiters -- finish -3651> 2014-05-19 18:37:21.497012 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3650> 2014-05-19 18:37:21.629542 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:24.687023) v2 ==== 47+0+0 (1951802779 0 0) 0x662bdc0 con 0x61ae420 -3649> 2014-05-19 18:37:21.629577 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:24.687023) v2 ==== 47+0+0 (1951802779 0 0) 0x860e000 con 0x618fe40 -3648> 2014-05-19 18:37:21.629591 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:24.687023) v2 -- ?+0 0x5dfe8c0 con 0x61ae420 -3647> 2014-05-19 18:37:21.629622 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3646> 2014-05-19 18:37:21.629748 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:24.687023) v2 -- ?+0 0x70a8540 con 0x618fe40 -3645> 2014-05-19 18:37:21.629786 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3644> 2014-05-19 18:37:21.678175 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:26.766223) v2 ==== 47+0+0 (2692733641 0 0) 0x60e0700 con 0x61b0160 -3643> 2014-05-19 18:37:21.678176 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:26.766223) v2 ==== 47+0+0 (2692733641 0 0) 0x36f6000 con 0x31e1600 -3642> 2014-05-19 18:37:21.678227 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:26.766223) v2 -- ?+0 0x860e000 con 0x61b0160 -3641> 2014-05-19 18:37:21.678265 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3640> 2014-05-19 18:37:21.678285 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:26.766223) v2 -- ?+0 0x662bdc0 con 0x31e1600 -3639> 2014-05-19 18:37:21.678317 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3638> 2014-05-19 18:37:21.779742 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:21.779545) v2 ==== 47+0+0 (2566914191 0 0) 0x64ae1c0 con 0x4cb82c0 -3637> 2014-05-19 18:37:21.779740 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:21.779545) v2 ==== 47+0+0 (2566914191 0 0) 0x7947dc0 con 0x4cb8160 -3636> 2014-05-19 18:37:21.779790 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:21.779545) v2 -- ?+0 0x60e0700 con 0x4cb82c0 -3635> 2014-05-19 18:37:21.779827 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3634> 2014-05-19 18:37:21.779858 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:21.779545) v2 -- ?+0 0x36f6000 con 0x4cb8160 -3633> 2014-05-19 18:37:21.779888 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3632> 2014-05-19 18:37:22.497066 7fd04f907700 5 osd.0 21 tick -3631> 2014-05-19 18:37:22.497084 7fd04f907700 7 osd.0 21 do_mon_report -3630> 2014-05-19 18:37:22.497089 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -3629> 2014-05-19 18:37:22.497095 7fd04f907700 20 osd.0 21 send_pg_stats -3628> 2014-05-19 18:37:22.497145 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3627> 2014-05-19 18:37:22.497138 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.03 < max 0.5 = yes -3626> 2014-05-19 18:37:22.497156 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -3625> 2014-05-19 18:37:22.497161 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 285.221 < min (86400 seconds) -3624> 2014-05-19 18:37:22.497197 7fd04f907700 20 osd.0 21 sched_scrub done -3623> 2014-05-19 18:37:22.497203 7fd04f907700 10 osd.0 21 do_waiters -- start -3622> 2014-05-19 18:37:22.497206 7fd04f907700 10 osd.0 21 do_waiters -- finish -3621> 2014-05-19 18:37:22.776542 7fd04e905700 20 KeyValueStore::op_tp worker waiting -3620> 2014-05-19 18:37:22.776550 7fd043ebd700 20 OSD::op_tp worker waiting -3619> 2014-05-19 18:37:22.948760 7fd0396a8700 20 OSD::command_tp worker waiting -3618> 2014-05-19 18:37:23.343233 7fd041eb9700 20 OSD::op_tp worker waiting -3617> 2014-05-19 18:37:23.343257 7fd0436bc700 20 OSD::op_tp worker waiting -3616> 2014-05-19 18:37:23.343279 7fd03d6b0700 20 OSD::op_tp worker waiting -3615> 2014-05-19 18:37:23.343294 7fd03e6b2700 20 OSD::op_tp worker waiting -3614> 2014-05-19 18:37:23.343313 7fd03c6ae700 20 OSD::op_tp worker waiting -3613> 2014-05-19 18:37:23.343358 7fd03bead700 20 OSD::op_tp worker waiting -3612> 2014-05-19 18:37:23.343429 7fd03f6b4700 20 OSD::op_tp worker waiting -3611> 2014-05-19 18:37:23.357902 7fd03feb5700 20 OSD::op_tp worker waiting -3610> 2014-05-19 18:37:23.357928 7fd0446be700 20 OSD::op_tp worker waiting -3609> 2014-05-19 18:37:23.364743 7fd040eb7700 20 OSD::op_tp worker waiting -3608> 2014-05-19 18:37:23.370288 7fd0406b6700 20 OSD::op_tp worker waiting -3607> 2014-05-19 18:37:23.391751 7fd039ea9700 20 OSD::disk_tp worker waiting -3606> 2014-05-19 18:37:23.395340 7fd03aeab700 20 OSD::op_tp worker waiting -3605> 2014-05-19 18:37:23.395367 7fd03eeb3700 20 OSD::op_tp worker waiting -3604> 2014-05-19 18:37:23.413381 7fd042ebb700 20 OSD::op_tp worker waiting -3603> 2014-05-19 18:37:23.413406 7fd03deb1700 20 OSD::op_tp worker waiting -3602> 2014-05-19 18:37:23.422087 7fd04e104700 20 KeyValueStore::op_tp worker waiting -3601> 2014-05-19 18:37:23.422094 7fd03ceaf700 20 OSD::op_tp worker waiting -3600> 2014-05-19 18:37:23.422114 7fd0426ba700 20 OSD::op_tp worker waiting -3599> 2014-05-19 18:37:23.422134 7fd03b6ac700 20 OSD::op_tp worker waiting -3598> 2014-05-19 18:37:23.422144 7fd0416b8700 20 OSD::op_tp worker waiting -3597> 2014-05-19 18:37:23.497294 7fd04f907700 5 osd.0 21 tick -3596> 2014-05-19 18:37:23.497314 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3595> 2014-05-19 18:37:23.497319 7fd04f907700 10 osd.0 21 do_waiters -- start -3594> 2014-05-19 18:37:23.497321 7fd04f907700 10 osd.0 21 do_waiters -- finish -3593> 2014-05-19 18:37:23.497351 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3592> 2014-05-19 18:37:23.694453 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:23.694279) v2 ==== 47+0+0 (2211725587 0 0) 0x55c2380 con 0x4cb86e0 -3591> 2014-05-19 18:37:23.694476 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:23.694279) v2 ==== 47+0+0 (2211725587 0 0) 0x656c380 con 0x4cb8420 -3590> 2014-05-19 18:37:23.694503 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:23.694279) v2 -- ?+0 0x7947dc0 con 0x4cb86e0 -3589> 2014-05-19 18:37:23.694538 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3588> 2014-05-19 18:37:23.694582 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:23.694279) v2 -- ?+0 0x64ae1c0 con 0x4cb8420 -3587> 2014-05-19 18:37:23.694619 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3586> 2014-05-19 18:37:24.497398 7fd04f907700 5 osd.0 21 tick -3585> 2014-05-19 18:37:24.497418 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3584> 2014-05-19 18:37:24.497423 7fd04f907700 10 osd.0 21 do_waiters -- start -3583> 2014-05-19 18:37:24.497425 7fd04f907700 10 osd.0 21 do_waiters -- finish -3582> 2014-05-19 18:37:24.497466 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3581> 2014-05-19 18:37:24.768240 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:27.825750) v2 ==== 47+0+0 (1961366503 0 0) 0x784c380 con 0x4cb9340 -3580> 2014-05-19 18:37:24.768282 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:27.825750) v2 -- ?+0 0x656c380 con 0x4cb9340 -3579> 2014-05-19 18:37:24.768313 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3578> 2014-05-19 18:37:24.768268 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:27.825750) v2 ==== 47+0+0 (1961366503 0 0) 0x72dfdc0 con 0x4cb8dc0 -3577> 2014-05-19 18:37:24.768340 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:27.825750) v2 -- ?+0 0x55c2380 con 0x4cb8dc0 -3576> 2014-05-19 18:37:24.768374 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3575> 2014-05-19 18:37:24.776640 7fd04e905700 20 KeyValueStore::op_tp worker waiting -3574> 2014-05-19 18:37:24.776643 7fd043ebd700 20 OSD::op_tp worker waiting -3573> 2014-05-19 18:37:24.893697 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:29.981875) v2 ==== 47+0+0 (3477178441 0 0) 0x6543880 con 0x5f5b760 -3572> 2014-05-19 18:37:24.893699 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:29.981875) v2 ==== 47+0+0 (3477178441 0 0) 0x784ca80 con 0x5b21600 -3571> 2014-05-19 18:37:24.893756 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:29.981875) v2 -- ?+0 0x784c380 con 0x5f5b760 -3570> 2014-05-19 18:37:24.893791 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3569> 2014-05-19 18:37:24.893821 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:29.981875) v2 -- ?+0 0x72dfdc0 con 0x5b21600 -3568> 2014-05-19 18:37:24.893862 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3567> 2014-05-19 18:37:24.948851 7fd0396a8700 20 OSD::command_tp worker waiting -3566> 2014-05-19 18:37:25.343328 7fd0436bc700 20 OSD::op_tp worker waiting -3565> 2014-05-19 18:37:25.343351 7fd041eb9700 20 OSD::op_tp worker waiting -3564> 2014-05-19 18:37:25.343366 7fd03d6b0700 20 OSD::op_tp worker waiting -3563> 2014-05-19 18:37:25.343388 7fd03c6ae700 20 OSD::op_tp worker waiting -3562> 2014-05-19 18:37:25.343409 7fd03e6b2700 20 OSD::op_tp worker waiting -3561> 2014-05-19 18:37:25.343429 7fd03bead700 20 OSD::op_tp worker waiting -3560> 2014-05-19 18:37:25.343492 7fd03f6b4700 20 OSD::op_tp worker waiting -3559> 2014-05-19 18:37:25.357978 7fd03feb5700 20 OSD::op_tp worker waiting -3558> 2014-05-19 18:37:25.358006 7fd0446be700 20 OSD::op_tp worker waiting -3557> 2014-05-19 18:37:25.364840 7fd040eb7700 20 OSD::op_tp worker waiting -3556> 2014-05-19 18:37:25.370385 7fd0406b6700 20 OSD::op_tp worker waiting -3555> 2014-05-19 18:37:25.391831 7fd039ea9700 20 OSD::disk_tp worker waiting -3554> 2014-05-19 18:37:25.395438 7fd03aeab700 20 OSD::op_tp worker waiting -3553> 2014-05-19 18:37:25.395462 7fd03eeb3700 20 OSD::op_tp worker waiting -3552> 2014-05-19 18:37:25.413464 7fd042ebb700 20 OSD::op_tp worker waiting -3551> 2014-05-19 18:37:25.413485 7fd03deb1700 20 OSD::op_tp worker waiting -3550> 2014-05-19 18:37:25.422182 7fd03ceaf700 20 OSD::op_tp worker waiting -3549> 2014-05-19 18:37:25.422181 7fd04e104700 20 KeyValueStore::op_tp worker waiting -3548> 2014-05-19 18:37:25.422214 7fd0426ba700 20 OSD::op_tp worker waiting -3547> 2014-05-19 18:37:25.422233 7fd03b6ac700 20 OSD::op_tp worker waiting -3546> 2014-05-19 18:37:25.422254 7fd0416b8700 20 OSD::op_tp worker waiting -3545> 2014-05-19 18:37:25.497509 7fd04f907700 5 osd.0 21 tick -3544> 2014-05-19 18:37:25.497530 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3543> 2014-05-19 18:37:25.497535 7fd04f907700 10 osd.0 21 do_waiters -- start -3542> 2014-05-19 18:37:25.497537 7fd04f907700 10 osd.0 21 do_waiters -- finish -3541> 2014-05-19 18:37:25.497578 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3540> 2014-05-19 18:37:25.764550 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -3539> 2014-05-19 18:37:25.764568 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -3538> 2014-05-19 18:37:25.764600 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x662ba40 con 0x3622580 -3537> 2014-05-19 18:37:25.764646 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x5f0b180 con 0x3622840 -3536> 2014-05-19 18:37:25.764675 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x59bcc40 con 0x3622b00 -3535> 2014-05-19 18:37:25.764702 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x6c4a700 con 0x3622dc0 -3534> 2014-05-19 18:37:25.764733 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x772d180 con 0x30594a0 -3533> 2014-05-19 18:37:25.764762 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x60a0700 con 0x3622f20 -3532> 2014-05-19 18:37:25.764793 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x7733c00 con 0x5e7ef20 -3531> 2014-05-19 18:37:25.764819 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x7a5ea80 con 0x5e7f1e0 -3530> 2014-05-19 18:37:25.764845 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x725d880 con 0x5e7e2c0 -3529> 2014-05-19 18:37:25.764872 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x5cbcfc0 con 0x5d7bb80 -3528> 2014-05-19 18:37:25.764898 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x662b6c0 con 0x5e7ec60 -3527> 2014-05-19 18:37:25.764930 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x6c4b180 con 0x531b760 -3526> 2014-05-19 18:37:25.764956 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x7733a40 con 0x53bba20 -3525> 2014-05-19 18:37:25.764983 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x656cfc0 con 0x51cac60 -3524> 2014-05-19 18:37:25.764988 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x60a0fc0 con 0x3622840 -3523> 2014-05-19 18:37:25.765013 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x7aa4540 con 0x5bd66e0 -3522> 2014-05-19 18:37:25.765038 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:25.764598) v2 -- ?+0 0x6543500 con 0x5bd6840 -3521> 2014-05-19 18:37:25.765126 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3520> 2014-05-19 18:37:25.765166 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x6b71a40 con 0x3622b00 -3519> 2014-05-19 18:37:25.765197 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3518> 2014-05-19 18:37:25.765218 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x6505500 con 0x3622dc0 -3517> 2014-05-19 18:37:25.765247 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3516> 2014-05-19 18:37:25.765258 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x7a5f340 con 0x30594a0 -3515> 2014-05-19 18:37:25.765299 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3514> 2014-05-19 18:37:25.765333 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x55f48c0 con 0x3622580 -3513> 2014-05-19 18:37:25.765383 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3512> 2014-05-19 18:37:25.765406 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x7971dc0 con 0x5e7ef20 -3511> 2014-05-19 18:37:25.765435 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3510> 2014-05-19 18:37:25.765446 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x83881c0 con 0x5e7f1e0 -3509> 2014-05-19 18:37:25.765466 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3508> 2014-05-19 18:37:25.765472 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x5f06000 con 0x5e7e2c0 -3507> 2014-05-19 18:37:25.765508 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3506> 2014-05-19 18:37:25.765516 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x79988c0 con 0x5d7bb80 -3505> 2014-05-19 18:37:25.765531 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3504> 2014-05-19 18:37:25.765536 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x73c2000 con 0x531b760 -3503> 2014-05-19 18:37:25.765550 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3502> 2014-05-19 18:37:25.765558 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x66f8700 con 0x53bba20 -3501> 2014-05-19 18:37:25.765572 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3500> 2014-05-19 18:37:25.765578 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x71cb340 con 0x51cac60 -3499> 2014-05-19 18:37:25.765591 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3498> 2014-05-19 18:37:25.765599 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x7735180 con 0x5bd66e0 -3497> 2014-05-19 18:37:25.765613 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3496> 2014-05-19 18:37:25.765624 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 85 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x656c8c0 con 0x5bd6840 -3495> 2014-05-19 18:37:25.765638 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3494> 2014-05-19 18:37:25.765646 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x682fc00 con 0x3622f20 -3493> 2014-05-19 18:37:25.765660 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3492> 2014-05-19 18:37:25.765668 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:25.764598) v2 ==== 47+0+0 (2190110211 0 0) 0x798e700 con 0x5e7ec60 -3491> 2014-05-19 18:37:25.765685 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3490> 2014-05-19 18:37:25.818478 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:30.906675) v2 ==== 47+0+0 (3682698657 0 0) 0x5f0aa80 con 0x3094160 -3489> 2014-05-19 18:37:25.818489 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 82 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:30.906675) v2 ==== 47+0+0 (3682698657 0 0) 0x64aec40 con 0x30946e0 -3488> 2014-05-19 18:37:25.818518 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:30.906675) v2 -- ?+0 0x6543880 con 0x3094160 -3487> 2014-05-19 18:37:25.818553 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3486> 2014-05-19 18:37:25.818578 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:30.906675) v2 -- ?+0 0x784ca80 con 0x30946e0 -3485> 2014-05-19 18:37:25.818609 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3484> 2014-05-19 18:37:26.497619 7fd04f907700 5 osd.0 21 tick -3483> 2014-05-19 18:37:26.497642 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3482> 2014-05-19 18:37:26.497649 7fd04f907700 10 osd.0 21 do_waiters -- start -3481> 2014-05-19 18:37:26.497653 7fd04f907700 10 osd.0 21 do_waiters -- finish -3480> 2014-05-19 18:37:26.497680 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3479> 2014-05-19 18:37:26.699119 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:29.756636) v2 ==== 47+0+0 (3121470649 0 0) 0x5dfee00 con 0x61ae160 -3478> 2014-05-19 18:37:26.699125 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:29.756636) v2 ==== 47+0+0 (3121470649 0 0) 0x70a8e00 con 0x61b0420 -3477> 2014-05-19 18:37:26.699168 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:29.756636) v2 -- ?+0 0x64aec40 con 0x61ae160 -3476> 2014-05-19 18:37:26.699240 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3475> 2014-05-19 18:37:26.699297 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:29.756636) v2 -- ?+0 0x5f0aa80 con 0x61b0420 -3474> 2014-05-19 18:37:26.699340 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3473> 2014-05-19 18:37:26.776724 7fd04e905700 20 KeyValueStore::op_tp worker waiting -3472> 2014-05-19 18:37:26.776727 7fd043ebd700 20 OSD::op_tp worker waiting -3471> 2014-05-19 18:37:26.948946 7fd0396a8700 20 OSD::command_tp worker waiting -3470> 2014-05-19 18:37:27.343426 7fd0436bc700 20 OSD::op_tp worker waiting -3469> 2014-05-19 18:37:27.343453 7fd041eb9700 20 OSD::op_tp worker waiting -3468> 2014-05-19 18:37:27.343473 7fd03d6b0700 20 OSD::op_tp worker waiting -3467> 2014-05-19 18:37:27.343491 7fd03c6ae700 20 OSD::op_tp worker waiting -3466> 2014-05-19 18:37:27.343509 7fd03e6b2700 20 OSD::op_tp worker waiting -3465> 2014-05-19 18:37:27.343537 7fd03bead700 20 OSD::op_tp worker waiting -3464> 2014-05-19 18:37:27.343556 7fd03f6b4700 20 OSD::op_tp worker waiting -3463> 2014-05-19 18:37:27.358060 7fd03feb5700 20 OSD::op_tp worker waiting -3462> 2014-05-19 18:37:27.358083 7fd0446be700 20 OSD::op_tp worker waiting -3461> 2014-05-19 18:37:27.364938 7fd040eb7700 20 OSD::op_tp worker waiting -3460> 2014-05-19 18:37:27.370482 7fd0406b6700 20 OSD::op_tp worker waiting -3459> 2014-05-19 18:37:27.391912 7fd039ea9700 20 OSD::disk_tp worker waiting -3458> 2014-05-19 18:37:27.395523 7fd03aeab700 20 OSD::op_tp worker waiting -3457> 2014-05-19 18:37:27.395542 7fd03eeb3700 20 OSD::op_tp worker waiting -3456> 2014-05-19 18:37:27.413547 7fd042ebb700 20 OSD::op_tp worker waiting -3455> 2014-05-19 18:37:27.413567 7fd03deb1700 20 OSD::op_tp worker waiting -3454> 2014-05-19 18:37:27.422264 7fd03ceaf700 20 OSD::op_tp worker waiting -3453> 2014-05-19 18:37:27.422272 7fd04e104700 20 KeyValueStore::op_tp worker waiting -3452> 2014-05-19 18:37:27.422281 7fd0426ba700 20 OSD::op_tp worker waiting -3451> 2014-05-19 18:37:27.422298 7fd03b6ac700 20 OSD::op_tp worker waiting -3450> 2014-05-19 18:37:27.422334 7fd0416b8700 20 OSD::op_tp worker waiting -3449> 2014-05-19 18:37:27.497726 7fd04f907700 5 osd.0 21 tick -3448> 2014-05-19 18:37:27.497743 7fd04f907700 7 osd.0 21 do_mon_report -3447> 2014-05-19 18:37:27.497747 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -3446> 2014-05-19 18:37:27.497751 7fd04f907700 20 osd.0 21 send_pg_stats -3445> 2014-05-19 18:37:27.497753 7fd04f907700 10 osd.0 21 send_pg_stats - 0 pgs updated -3444> 2014-05-19 18:37:27.497758 7fd04f907700 10 monclient: _send_mon_message to mon.a at 172.16.96.201:6789/0 -3443> 2014-05-19 18:37:27.497766 7fd04f907700 1 -- 172.16.96.201:6800/3074 --> 172.16.96.201:6789/0 -- pg_stats(0 pgs tid 8 v 0) v1 -- ?+0 0x6b9af00 con 0x2e3edc0 -3442> 2014-05-19 18:37:27.497786 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3441> 2014-05-19 18:37:27.497792 7fd04f907700 10 osd.0 21 do_waiters -- start -3440> 2014-05-19 18:37:27.497795 7fd04f907700 10 osd.0 21 do_waiters -- finish -3439> 2014-05-19 18:37:27.497823 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3438> 2014-05-19 18:37:27.498182 7fd04aecb700 1 -- 172.16.96.201:6800/3074 <== mon.0 172.16.96.201:6789/0 45 ==== pg_stats_ack(0 pgs tid 8) v1 ==== 4+0+0 (0 0 0) 0x7890380 con 0x2e3edc0 -3437> 2014-05-19 18:37:27.498210 7fd04aecb700 10 osd.0 21 do_waiters -- start -3436> 2014-05-19 18:37:27.498214 7fd04aecb700 10 osd.0 21 do_waiters -- finish -3435> 2014-05-19 18:37:27.498216 7fd04aecb700 20 osd.0 21 _dispatch 0x7890380 pg_stats_ack(0 pgs tid 8) v1 -3434> 2014-05-19 18:37:27.498219 7fd04aecb700 10 osd.0 21 handle_pg_stats_ack -3433> 2014-05-19 18:37:27.498221 7fd04aecb700 10 osd.0 21 do_waiters -- start -3432> 2014-05-19 18:37:27.498223 7fd04aecb700 10 osd.0 21 do_waiters -- finish -3431> 2014-05-19 18:37:27.529949 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:30.587484) v2 ==== 47+0+0 (218431714 0 0) 0x5b281c0 con 0x618fe40 -3430> 2014-05-19 18:37:27.529959 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:30.587484) v2 ==== 47+0+0 (218431714 0 0) 0x7746fc0 con 0x61ae420 -3429> 2014-05-19 18:37:27.530000 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:30.587484) v2 -- ?+0 0x70a8e00 con 0x618fe40 -3428> 2014-05-19 18:37:27.530034 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3427> 2014-05-19 18:37:27.530054 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:30.587484) v2 -- ?+0 0x5dfee00 con 0x61ae420 -3426> 2014-05-19 18:37:27.530088 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3425> 2014-05-19 18:37:27.578556 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:32.666844) v2 ==== 47+0+0 (1023896258 0 0) 0x68ab500 con 0x61b0160 -3424> 2014-05-19 18:37:27.578594 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:32.666844) v2 -- ?+0 0x5b281c0 con 0x61b0160 -3423> 2014-05-19 18:37:27.578590 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:32.666844) v2 ==== 47+0+0 (1023896258 0 0) 0x64d3180 con 0x31e1600 -3422> 2014-05-19 18:37:27.578618 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3421> 2014-05-19 18:37:27.578656 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:32.666844) v2 -- ?+0 0x7746fc0 con 0x31e1600 -3420> 2014-05-19 18:37:27.578689 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3419> 2014-05-19 18:37:27.680282 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:27.680098) v2 ==== 47+0+0 (2453715655 0 0) 0x6aa5500 con 0x4cb82c0 -3418> 2014-05-19 18:37:27.680343 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:27.680098) v2 -- ?+0 0x68ab500 con 0x4cb82c0 -3417> 2014-05-19 18:37:27.680286 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:27.680098) v2 ==== 47+0+0 (2453715655 0 0) 0x2ee08c0 con 0x4cb8160 -3416> 2014-05-19 18:37:27.680383 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3415> 2014-05-19 18:37:27.680477 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:27.680098) v2 -- ?+0 0x64d3180 con 0x4cb8160 -3414> 2014-05-19 18:37:27.680603 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3413> 2014-05-19 18:37:28.497881 7fd04f907700 5 osd.0 21 tick -3412> 2014-05-19 18:37:28.497903 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3411> 2014-05-19 18:37:28.497907 7fd04f907700 10 osd.0 21 do_waiters -- start -3410> 2014-05-19 18:37:28.497910 7fd04f907700 10 osd.0 21 do_waiters -- finish -3409> 2014-05-19 18:37:28.497950 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3408> 2014-05-19 18:37:28.776824 7fd04e905700 20 KeyValueStore::op_tp worker waiting -3407> 2014-05-19 18:37:28.776828 7fd043ebd700 20 OSD::op_tp worker waiting -3406> 2014-05-19 18:37:28.949038 7fd0396a8700 20 OSD::command_tp worker waiting -3405> 2014-05-19 18:37:29.343479 7fd0436bc700 20 OSD::op_tp worker waiting -3404> 2014-05-19 18:37:29.343525 7fd03c6ae700 20 OSD::op_tp worker waiting -3403> 2014-05-19 18:37:29.343543 7fd03d6b0700 20 OSD::op_tp worker waiting -3402> 2014-05-19 18:37:29.343557 7fd041eb9700 20 OSD::op_tp worker waiting -3401> 2014-05-19 18:37:29.343582 7fd03e6b2700 20 OSD::op_tp worker waiting -3400> 2014-05-19 18:37:29.343619 7fd03f6b4700 20 OSD::op_tp worker waiting -3399> 2014-05-19 18:37:29.343637 7fd03bead700 20 OSD::op_tp worker waiting -3398> 2014-05-19 18:37:29.358116 7fd03feb5700 20 OSD::op_tp worker waiting -3397> 2014-05-19 18:37:29.358160 7fd0446be700 20 OSD::op_tp worker waiting -3396> 2014-05-19 18:37:29.365037 7fd040eb7700 20 OSD::op_tp worker waiting -3395> 2014-05-19 18:37:29.370579 7fd0406b6700 20 OSD::op_tp worker waiting -3394> 2014-05-19 18:37:29.392002 7fd039ea9700 20 OSD::disk_tp worker waiting -3393> 2014-05-19 18:37:29.395608 7fd03aeab700 20 OSD::op_tp worker waiting -3392> 2014-05-19 18:37:29.395627 7fd03eeb3700 20 OSD::op_tp worker waiting -3391> 2014-05-19 18:37:29.413630 7fd042ebb700 20 OSD::op_tp worker waiting -3390> 2014-05-19 18:37:29.413650 7fd03deb1700 20 OSD::op_tp worker waiting -3389> 2014-05-19 18:37:29.422360 7fd04e104700 20 KeyValueStore::op_tp worker waiting -3388> 2014-05-19 18:37:29.422365 7fd03ceaf700 20 OSD::op_tp worker waiting -3387> 2014-05-19 18:37:29.422410 7fd0416b8700 20 OSD::op_tp worker waiting -3386> 2014-05-19 18:37:29.422432 7fd0426ba700 20 OSD::op_tp worker waiting -3385> 2014-05-19 18:37:29.422456 7fd03b6ac700 20 OSD::op_tp worker waiting -3384> 2014-05-19 18:37:29.497994 7fd04f907700 5 osd.0 21 tick -3383> 2014-05-19 18:37:29.498015 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3382> 2014-05-19 18:37:29.498019 7fd04f907700 10 osd.0 21 do_waiters -- start -3381> 2014-05-19 18:37:29.498022 7fd04f907700 10 osd.0 21 do_waiters -- finish -3380> 2014-05-19 18:37:29.498063 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3379> 2014-05-19 18:37:29.595036 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:29.594821) v2 ==== 47+0+0 (2032275684 0 0) 0x64a7a40 con 0x4cb8420 -3378> 2014-05-19 18:37:29.595012 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:29.594821) v2 ==== 47+0+0 (2032275684 0 0) 0x55c28c0 con 0x4cb86e0 -3377> 2014-05-19 18:37:29.595081 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:29.594821) v2 -- ?+0 0x6aa5500 con 0x4cb8420 -3376> 2014-05-19 18:37:29.595129 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3375> 2014-05-19 18:37:29.595154 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:29.594821) v2 -- ?+0 0x2ee08c0 con 0x4cb86e0 -3374> 2014-05-19 18:37:29.595194 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3373> 2014-05-19 18:37:29.907866 7fd0456c0700 10 monclient: tick -3372> 2014-05-19 18:37:29.907878 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:36:59.907877) -3371> 2014-05-19 18:37:29.907897 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:37:29.907897; renew after: 2014-05-19 18:38:19.906664) -- no -3370> 2014-05-19 18:37:30.498105 7fd04f907700 5 osd.0 21 tick -3369> 2014-05-19 18:37:30.498125 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3368> 2014-05-19 18:37:30.498130 7fd04f907700 10 osd.0 21 do_waiters -- start -3367> 2014-05-19 18:37:30.498132 7fd04f907700 10 osd.0 21 do_waiters -- finish -3366> 2014-05-19 18:37:30.498162 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3365> 2014-05-19 18:37:30.668660 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:33.726274) v2 ==== 47+0+0 (1125335533 0 0) 0x788b500 con 0x4cb8dc0 -3364> 2014-05-19 18:37:30.668691 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:33.726274) v2 ==== 47+0+0 (1125335533 0 0) 0x784d6c0 con 0x4cb9340 -3363> 2014-05-19 18:37:30.668714 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.726274) v2 -- ?+0 0x55c28c0 con 0x4cb8dc0 -3362> 2014-05-19 18:37:30.668778 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3361> 2014-05-19 18:37:30.668803 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.726274) v2 -- ?+0 0x64a7a40 con 0x4cb9340 -3360> 2014-05-19 18:37:30.668861 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3359> 2014-05-19 18:37:30.776924 7fd04e905700 20 KeyValueStore::op_tp worker waiting -3358> 2014-05-19 18:37:30.776926 7fd043ebd700 20 OSD::op_tp worker waiting -3357> 2014-05-19 18:37:30.793981 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:35.882372) v2 ==== 47+0+0 (327488341 0 0) 0x6543c00 con 0x5f5b760 -3356> 2014-05-19 18:37:30.793992 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:35.882372) v2 ==== 47+0+0 (327488341 0 0) 0x70cc1c0 con 0x5b21600 -3355> 2014-05-19 18:37:30.794032 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:35.882372) v2 -- ?+0 0x784d6c0 con 0x5f5b760 -3354> 2014-05-19 18:37:30.794060 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3353> 2014-05-19 18:37:30.794084 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:35.882372) v2 -- ?+0 0x788b500 con 0x5b21600 -3352> 2014-05-19 18:37:30.794125 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3351> 2014-05-19 18:37:30.949132 7fd0396a8700 20 OSD::command_tp worker waiting -3350> 2014-05-19 18:37:31.343552 7fd0436bc700 20 OSD::op_tp worker waiting -3349> 2014-05-19 18:37:31.343598 7fd03c6ae700 20 OSD::op_tp worker waiting -3348> 2014-05-19 18:37:31.343622 7fd041eb9700 20 OSD::op_tp worker waiting -3347> 2014-05-19 18:37:31.343634 7fd03d6b0700 20 OSD::op_tp worker waiting -3346> 2014-05-19 18:37:31.343663 7fd03e6b2700 20 OSD::op_tp worker waiting -3345> 2014-05-19 18:37:31.343706 7fd03bead700 20 OSD::op_tp worker waiting -3344> 2014-05-19 18:37:31.343718 7fd03f6b4700 20 OSD::op_tp worker waiting -3343> 2014-05-19 18:37:31.358193 7fd03feb5700 20 OSD::op_tp worker waiting -3342> 2014-05-19 18:37:31.358240 7fd0446be700 20 OSD::op_tp worker waiting -3341> 2014-05-19 18:37:31.365135 7fd040eb7700 20 OSD::op_tp worker waiting -3340> 2014-05-19 18:37:31.370660 7fd0406b6700 20 OSD::op_tp worker waiting -3339> 2014-05-19 18:37:31.392098 7fd039ea9700 20 OSD::disk_tp worker waiting -3338> 2014-05-19 18:37:31.395706 7fd03aeab700 20 OSD::op_tp worker waiting -3337> 2014-05-19 18:37:31.395733 7fd03eeb3700 20 OSD::op_tp worker waiting -3336> 2014-05-19 18:37:31.413730 7fd042ebb700 20 OSD::op_tp worker waiting -3335> 2014-05-19 18:37:31.413756 7fd03deb1700 20 OSD::op_tp worker waiting -3334> 2014-05-19 18:37:31.422470 7fd03ceaf700 20 OSD::op_tp worker waiting -3333> 2014-05-19 18:37:31.422470 7fd04e104700 20 KeyValueStore::op_tp worker waiting -3332> 2014-05-19 18:37:31.422508 7fd0416b8700 20 OSD::op_tp worker waiting -3331> 2014-05-19 18:37:31.422537 7fd03b6ac700 20 OSD::op_tp worker waiting -3330> 2014-05-19 18:37:31.422569 7fd0426ba700 20 OSD::op_tp worker waiting -3329> 2014-05-19 18:37:31.498218 7fd04f907700 5 osd.0 21 tick -3328> 2014-05-19 18:37:31.498239 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3327> 2014-05-19 18:37:31.498243 7fd04f907700 10 osd.0 21 do_waiters -- start -3326> 2014-05-19 18:37:31.498245 7fd04f907700 10 osd.0 21 do_waiters -- finish -3325> 2014-05-19 18:37:31.498287 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3324> 2014-05-19 18:37:31.665177 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -3323> 2014-05-19 18:37:31.665195 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -3322> 2014-05-19 18:37:31.665203 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x66f9dc0 con 0x3622580 -3321> 2014-05-19 18:37:31.665228 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x60081c0 con 0x3622840 -3320> 2014-05-19 18:37:31.665267 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x5f0a380 con 0x3622b00 -3319> 2014-05-19 18:37:31.665319 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x692dc00 con 0x3622dc0 -3318> 2014-05-19 18:37:31.665349 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x8388540 con 0x30594a0 -3317> 2014-05-19 18:37:31.665375 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x772cc40 con 0x3622f20 -3316> 2014-05-19 18:37:31.665463 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x5f328c0 con 0x5e7ef20 -3315> 2014-05-19 18:37:31.665490 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x60a1180 con 0x5e7f1e0 -3314> 2014-05-19 18:37:31.665519 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x6b71500 con 0x5e7e2c0 -3313> 2014-05-19 18:37:31.665545 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x55f4fc0 con 0x5d7bb80 -3312> 2014-05-19 18:37:31.665575 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x77321c0 con 0x5e7ec60 -3311> 2014-05-19 18:37:31.665600 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x6d616c0 con 0x3622b00 -3310> 2014-05-19 18:37:31.665609 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x656c700 con 0x531b760 -3309> 2014-05-19 18:37:31.665651 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x77341c0 con 0x53bba20 -3308> 2014-05-19 18:37:31.665680 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x60f6700 con 0x51cac60 -3307> 2014-05-19 18:37:31.665706 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x66f9500 con 0x5bd66e0 -3306> 2014-05-19 18:37:31.665733 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:31.665201) v2 -- ?+0 0x60096c0 con 0x5bd6840 -3305> 2014-05-19 18:37:31.665779 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3304> 2014-05-19 18:37:31.665796 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x55f4380 con 0x3622580 -3303> 2014-05-19 18:37:31.665822 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3302> 2014-05-19 18:37:31.665833 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x7a37180 con 0x3622840 -3301> 2014-05-19 18:37:31.665849 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3300> 2014-05-19 18:37:31.665855 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x6504700 con 0x3622dc0 -3299> 2014-05-19 18:37:31.665882 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3298> 2014-05-19 18:37:31.665922 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x8388a80 con 0x5e7f1e0 -3297> 2014-05-19 18:37:31.665970 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3296> 2014-05-19 18:37:31.665989 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x7a5e1c0 con 0x30594a0 -3295> 2014-05-19 18:37:31.666015 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3294> 2014-05-19 18:37:31.666034 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x682e540 con 0x3622f20 -3293> 2014-05-19 18:37:31.666053 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3292> 2014-05-19 18:37:31.666059 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x7970540 con 0x5e7ef20 -3291> 2014-05-19 18:37:31.666097 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3290> 2014-05-19 18:37:31.666116 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x5f07340 con 0x5e7e2c0 -3289> 2014-05-19 18:37:31.666150 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3288> 2014-05-19 18:37:31.666172 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x7998700 con 0x5d7bb80 -3287> 2014-05-19 18:37:31.666194 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3286> 2014-05-19 18:37:31.666204 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x612ca80 con 0x5e7ec60 -3285> 2014-05-19 18:37:31.666219 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3284> 2014-05-19 18:37:31.666224 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x73c2540 con 0x531b760 -3283> 2014-05-19 18:37:31.666238 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3282> 2014-05-19 18:37:31.666247 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x5c04000 con 0x53bba20 -3281> 2014-05-19 18:37:31.666264 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3280> 2014-05-19 18:37:31.666272 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x5519880 con 0x51cac60 -3279> 2014-05-19 18:37:31.666286 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3278> 2014-05-19 18:37:31.666293 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x7734fc0 con 0x5bd66e0 -3277> 2014-05-19 18:37:31.666307 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3276> 2014-05-19 18:37:31.666315 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 86 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:31.665201) v2 ==== 47+0+0 (3396827601 0 0) 0x7aa41c0 con 0x5bd6840 -3275> 2014-05-19 18:37:31.666330 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3274> 2014-05-19 18:37:31.718697 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:36.807169) v2 ==== 47+0+0 (3176112301 0 0) 0x77e81c0 con 0x3094160 -3273> 2014-05-19 18:37:31.718747 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:36.807169) v2 -- ?+0 0x6543c00 con 0x3094160 -3272> 2014-05-19 18:37:31.718730 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 83 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:36.807169) v2 ==== 47+0+0 (3176112301 0 0) 0x70ea540 con 0x30946e0 -3271> 2014-05-19 18:37:31.718790 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3270> 2014-05-19 18:37:31.718843 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:36.807169) v2 -- ?+0 0x70cc1c0 con 0x30946e0 -3269> 2014-05-19 18:37:31.718879 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3268> 2014-05-19 18:37:32.498303 7fd04f907700 5 osd.0 21 tick -3267> 2014-05-19 18:37:32.498320 7fd04f907700 7 osd.0 21 do_mon_report -3266> 2014-05-19 18:37:32.498323 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -3265> 2014-05-19 18:37:32.498327 7fd04f907700 20 osd.0 21 send_pg_stats -3264> 2014-05-19 18:37:32.498335 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3263> 2014-05-19 18:37:32.498338 7fd04f907700 10 osd.0 21 do_waiters -- start -3262> 2014-05-19 18:37:32.498340 7fd04f907700 10 osd.0 21 do_waiters -- finish -3261> 2014-05-19 18:37:32.498371 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3260> 2014-05-19 18:37:32.599500 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:35.657115) v2 ==== 47+0+0 (459679352 0 0) 0x70eb500 con 0x61ae160 -3259> 2014-05-19 18:37:32.599527 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:35.657115) v2 ==== 47+0+0 (459679352 0 0) 0x64a7500 con 0x61b0420 -3258> 2014-05-19 18:37:32.599550 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:35.657115) v2 -- ?+0 0x70ea540 con 0x61ae160 -3257> 2014-05-19 18:37:32.599585 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3256> 2014-05-19 18:37:32.599607 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:35.657115) v2 -- ?+0 0x77e81c0 con 0x61b0420 -3255> 2014-05-19 18:37:32.599641 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3254> 2014-05-19 18:37:32.777019 7fd04e905700 20 KeyValueStore::op_tp worker waiting -3253> 2014-05-19 18:37:32.777020 7fd043ebd700 20 OSD::op_tp worker waiting -3252> 2014-05-19 18:37:32.949221 7fd0396a8700 20 OSD::command_tp worker waiting -3251> 2014-05-19 18:37:33.343633 7fd0436bc700 20 OSD::op_tp worker waiting -3250> 2014-05-19 18:37:33.343659 7fd03c6ae700 20 OSD::op_tp worker waiting -3249> 2014-05-19 18:37:33.343684 7fd041eb9700 20 OSD::op_tp worker waiting -3248> 2014-05-19 18:37:33.343708 7fd03d6b0700 20 OSD::op_tp worker waiting -3247> 2014-05-19 18:37:33.343721 7fd03e6b2700 20 OSD::op_tp worker waiting -3246> 2014-05-19 18:37:33.343765 7fd03bead700 20 OSD::op_tp worker waiting -3245> 2014-05-19 18:37:33.343783 7fd03f6b4700 20 OSD::op_tp worker waiting -3244> 2014-05-19 18:37:33.358253 7fd03feb5700 20 OSD::op_tp worker waiting -3243> 2014-05-19 18:37:33.358309 7fd0446be700 20 OSD::op_tp worker waiting -3242> 2014-05-19 18:37:33.365232 7fd040eb7700 20 OSD::op_tp worker waiting -3241> 2014-05-19 18:37:33.370757 7fd0406b6700 20 OSD::op_tp worker waiting -3240> 2014-05-19 18:37:33.392192 7fd039ea9700 20 OSD::disk_tp worker waiting -3239> 2014-05-19 18:37:33.395792 7fd03aeab700 20 OSD::op_tp worker waiting -3238> 2014-05-19 18:37:33.395812 7fd03eeb3700 20 OSD::op_tp worker waiting -3237> 2014-05-19 18:37:33.413812 7fd042ebb700 20 OSD::op_tp worker waiting -3236> 2014-05-19 18:37:33.413834 7fd03deb1700 20 OSD::op_tp worker waiting -3235> 2014-05-19 18:37:33.422574 7fd04e104700 20 KeyValueStore::op_tp worker waiting -3234> 2014-05-19 18:37:33.422577 7fd03ceaf700 20 OSD::op_tp worker waiting -3233> 2014-05-19 18:37:33.422613 7fd0416b8700 20 OSD::op_tp worker waiting -3232> 2014-05-19 18:37:33.422634 7fd0426ba700 20 OSD::op_tp worker waiting -3231> 2014-05-19 18:37:33.422649 7fd03b6ac700 20 OSD::op_tp worker waiting -3230> 2014-05-19 18:37:33.430420 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:36.488021) v2 ==== 47+0+0 (536610504 0 0) 0x7a36fc0 con 0x61ae420 -3229> 2014-05-19 18:37:33.430470 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:36.488021) v2 -- ?+0 0x70eb500 con 0x61ae420 -3228> 2014-05-19 18:37:33.430479 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:36.488021) v2 ==== 47+0+0 (536610504 0 0) 0x55c3500 con 0x618fe40 -3227> 2014-05-19 18:37:33.430548 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3226> 2014-05-19 18:37:33.430589 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:36.488021) v2 -- ?+0 0x64a7500 con 0x618fe40 -3225> 2014-05-19 18:37:33.430654 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3224> 2014-05-19 18:37:33.479000 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:38.567512) v2 ==== 47+0+0 (2731037548 0 0) 0x669e540 con 0x61b0160 -3223> 2014-05-19 18:37:33.479009 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:38.567512) v2 ==== 47+0+0 (2731037548 0 0) 0x5496000 con 0x31e1600 -3222> 2014-05-19 18:37:33.479057 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.567512) v2 -- ?+0 0x55c3500 con 0x61b0160 -3221> 2014-05-19 18:37:33.479094 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3220> 2014-05-19 18:37:33.479120 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.567512) v2 -- ?+0 0x7a36fc0 con 0x31e1600 -3219> 2014-05-19 18:37:33.479152 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3218> 2014-05-19 18:37:33.498412 7fd04f907700 5 osd.0 21 tick -3217> 2014-05-19 18:37:33.498433 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3216> 2014-05-19 18:37:33.498438 7fd04f907700 10 osd.0 21 do_waiters -- start -3215> 2014-05-19 18:37:33.498440 7fd04f907700 10 osd.0 21 do_waiters -- finish -3214> 2014-05-19 18:37:33.498469 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3213> 2014-05-19 18:37:33.580881 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:33.580685) v2 ==== 47+0+0 (391537540 0 0) 0x2ee1dc0 con 0x4cb8160 -3212> 2014-05-19 18:37:33.580889 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:33.580685) v2 ==== 47+0+0 (391537540 0 0) 0x6aa5880 con 0x4cb82c0 -3211> 2014-05-19 18:37:33.580927 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.580685) v2 -- ?+0 0x5496000 con 0x4cb8160 -3210> 2014-05-19 18:37:33.581014 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3209> 2014-05-19 18:37:33.581037 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.580685) v2 -- ?+0 0x669e540 con 0x4cb82c0 -3208> 2014-05-19 18:37:33.581075 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3207> 2014-05-19 18:37:33.965873 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -3206> 2014-05-19 18:37:33.965891 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -3205> 2014-05-19 18:37:33.965900 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x798f880 con 0x3622580 -3204> 2014-05-19 18:37:33.965925 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x7999dc0 con 0x3622840 -3203> 2014-05-19 18:37:33.965963 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x59bd500 con 0x3622b00 -3202> 2014-05-19 18:37:33.966001 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x8388700 con 0x3622dc0 -3201> 2014-05-19 18:37:33.966029 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x772ce00 con 0x30594a0 -3200> 2014-05-19 18:37:33.966059 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x5f7ac40 con 0x3622f20 -3199> 2014-05-19 18:37:33.966087 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x7a5e8c0 con 0x5e7ef20 -3198> 2014-05-19 18:37:33.966116 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x60a16c0 con 0x5e7f1e0 -3197> 2014-05-19 18:37:33.966143 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x649e540 con 0x5e7e2c0 -3196> 2014-05-19 18:37:33.966170 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x6b71c00 con 0x5d7bb80 -3195> 2014-05-19 18:37:33.966197 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x59bddc0 con 0x5e7ec60 -3194> 2014-05-19 18:37:33.966224 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x772d340 con 0x531b760 -3193> 2014-05-19 18:37:33.966253 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x7733dc0 con 0x53bba20 -3192> 2014-05-19 18:37:33.966281 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x7a5e000 con 0x51cac60 -3191> 2014-05-19 18:37:33.966304 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x656c000 con 0x5bd66e0 -3190> 2014-05-19 18:37:33.966328 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:33.965898) v2 -- ?+0 0x7735340 con 0x5bd6840 -3189> 2014-05-19 18:37:33.966322 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x74a76c0 con 0x3622580 -3188> 2014-05-19 18:37:33.966417 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3187> 2014-05-19 18:37:33.966427 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x7a37880 con 0x3622840 -3186> 2014-05-19 18:37:33.966453 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3185> 2014-05-19 18:37:33.966468 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x6d61180 con 0x3622b00 -3184> 2014-05-19 18:37:33.966485 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3183> 2014-05-19 18:37:33.966491 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x6504e00 con 0x3622dc0 -3182> 2014-05-19 18:37:33.966505 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3181> 2014-05-19 18:37:33.966538 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x7971500 con 0x5e7ef20 -3180> 2014-05-19 18:37:33.966580 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3179> 2014-05-19 18:37:33.966600 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x6009880 con 0x5e7e2c0 -3178> 2014-05-19 18:37:33.966634 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3177> 2014-05-19 18:37:33.966676 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x7998c40 con 0x5d7bb80 -3176> 2014-05-19 18:37:33.966712 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3175> 2014-05-19 18:37:33.966740 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x7747180 con 0x30594a0 -3174> 2014-05-19 18:37:33.966786 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3173> 2014-05-19 18:37:33.966796 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x3511180 con 0x3622f20 -3172> 2014-05-19 18:37:33.966813 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3171> 2014-05-19 18:37:33.966818 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x6e2ba40 con 0x5e7f1e0 -3170> 2014-05-19 18:37:33.966832 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3169> 2014-05-19 18:37:33.966840 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x612ce00 con 0x5e7ec60 -3168> 2014-05-19 18:37:33.966855 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3167> 2014-05-19 18:37:33.966863 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x73c3a40 con 0x531b760 -3166> 2014-05-19 18:37:33.966877 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3165> 2014-05-19 18:37:33.966885 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x5c05a40 con 0x53bba20 -3164> 2014-05-19 18:37:33.966901 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3163> 2014-05-19 18:37:33.966909 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x5518700 con 0x51cac60 -3162> 2014-05-19 18:37:33.966923 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3161> 2014-05-19 18:37:33.966931 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x7734540 con 0x5bd66e0 -3160> 2014-05-19 18:37:33.966945 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3159> 2014-05-19 18:37:33.966953 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 87 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:33.965898) v2 ==== 47+0+0 (3196577180 0 0) 0x7aa5880 con 0x5bd6840 -3158> 2014-05-19 18:37:33.966967 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3157> 2014-05-19 18:37:34.498508 7fd04f907700 5 osd.0 21 tick -3156> 2014-05-19 18:37:34.498567 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3155> 2014-05-19 18:37:34.498565 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes -3154> 2014-05-19 18:37:34.498596 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -3153> 2014-05-19 18:37:34.498601 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 297.223 < min (86400 seconds) -3152> 2014-05-19 18:37:34.498627 7fd04f907700 20 osd.0 21 sched_scrub done -3151> 2014-05-19 18:37:34.498632 7fd04f907700 10 osd.0 21 do_waiters -- start -3150> 2014-05-19 18:37:34.498636 7fd04f907700 10 osd.0 21 do_waiters -- finish -3149> 2014-05-19 18:37:34.777118 7fd04e905700 20 KeyValueStore::op_tp worker waiting -3148> 2014-05-19 18:37:34.777122 7fd043ebd700 20 OSD::op_tp worker waiting -3147> 2014-05-19 18:37:34.949301 7fd0396a8700 20 OSD::command_tp worker waiting -3146> 2014-05-19 18:37:35.343721 7fd0436bc700 20 OSD::op_tp worker waiting -3145> 2014-05-19 18:37:35.343749 7fd041eb9700 20 OSD::op_tp worker waiting -3144> 2014-05-19 18:37:35.343772 7fd03c6ae700 20 OSD::op_tp worker waiting -3143> 2014-05-19 18:37:35.343796 7fd03d6b0700 20 OSD::op_tp worker waiting -3142> 2014-05-19 18:37:35.343818 7fd03e6b2700 20 OSD::op_tp worker waiting -3141> 2014-05-19 18:37:35.343831 7fd03bead700 20 OSD::op_tp worker waiting -3140> 2014-05-19 18:37:35.343885 7fd03f6b4700 20 OSD::op_tp worker waiting -3139> 2014-05-19 18:37:35.358331 7fd03feb5700 20 OSD::op_tp worker waiting -3138> 2014-05-19 18:37:35.358382 7fd0446be700 20 OSD::op_tp worker waiting -3137> 2014-05-19 18:37:35.365328 7fd040eb7700 20 OSD::op_tp worker waiting -3136> 2014-05-19 18:37:35.370855 7fd0406b6700 20 OSD::op_tp worker waiting -3135> 2014-05-19 18:37:35.392288 7fd039ea9700 20 OSD::disk_tp worker waiting -3134> 2014-05-19 18:37:35.395875 7fd03aeab700 20 OSD::op_tp worker waiting -3133> 2014-05-19 18:37:35.395895 7fd03eeb3700 20 OSD::op_tp worker waiting -3132> 2014-05-19 18:37:35.413908 7fd042ebb700 20 OSD::op_tp worker waiting -3131> 2014-05-19 18:37:35.413931 7fd03deb1700 20 OSD::op_tp worker waiting -3130> 2014-05-19 18:37:35.422682 7fd04e104700 20 KeyValueStore::op_tp worker waiting -3129> 2014-05-19 18:37:35.422688 7fd03ceaf700 20 OSD::op_tp worker waiting -3128> 2014-05-19 18:37:35.422720 7fd0416b8700 20 OSD::op_tp worker waiting -3127> 2014-05-19 18:37:35.422743 7fd0426ba700 20 OSD::op_tp worker waiting -3126> 2014-05-19 18:37:35.422760 7fd03b6ac700 20 OSD::op_tp worker waiting -3125> 2014-05-19 18:37:35.495546 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:35.495365) v2 ==== 47+0+0 (1347956242 0 0) 0x6aa4700 con 0x4cb86e0 -3124> 2014-05-19 18:37:35.495593 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:35.495365) v2 -- ?+0 0x2ee1dc0 con 0x4cb86e0 -3123> 2014-05-19 18:37:35.495616 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3122> 2014-05-19 18:37:35.495630 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:35.495365) v2 ==== 47+0+0 (1347956242 0 0) 0x64a7dc0 con 0x4cb8420 -3121> 2014-05-19 18:37:35.495688 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:35.495365) v2 -- ?+0 0x6aa5880 con 0x4cb8420 -3120> 2014-05-19 18:37:35.495724 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3119> 2014-05-19 18:37:35.498702 7fd04f907700 5 osd.0 21 tick -3118> 2014-05-19 18:37:35.498718 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3117> 2014-05-19 18:37:35.498721 7fd04f907700 10 osd.0 21 do_waiters -- start -3116> 2014-05-19 18:37:35.498723 7fd04f907700 10 osd.0 21 do_waiters -- finish -3115> 2014-05-19 18:37:35.498757 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3114> 2014-05-19 18:37:35.730896 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:38.788488) v2 ==== 47+0+0 (2689648923 0 0) 0x8649880 con 0x618fe40 -3113> 2014-05-19 18:37:35.730938 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.788488) v2 -- ?+0 0x64a7dc0 con 0x618fe40 -3112> 2014-05-19 18:37:35.730972 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3111> 2014-05-19 18:37:35.730846 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:38.788488) v2 ==== 47+0+0 (2689648923 0 0) 0x77e8380 con 0x61ae420 -3110> 2014-05-19 18:37:35.730992 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.788488) v2 -- ?+0 0x6aa4700 con 0x61ae420 -3109> 2014-05-19 18:37:35.731035 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3108> 2014-05-19 18:37:35.881434 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:35.881257) v2 ==== 47+0+0 (4025922579 0 0) 0x6aa56c0 con 0x4cb82c0 -3107> 2014-05-19 18:37:35.881457 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:35.881257) v2 ==== 47+0+0 (4025922579 0 0) 0x2ee16c0 con 0x4cb8160 -3106> 2014-05-19 18:37:35.881475 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:35.881257) v2 -- ?+0 0x8649880 con 0x4cb82c0 -3105> 2014-05-19 18:37:35.881500 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3104> 2014-05-19 18:37:35.881555 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:35.881257) v2 -- ?+0 0x77e8380 con 0x4cb8160 -3103> 2014-05-19 18:37:35.881593 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3102> 2014-05-19 18:37:36.498786 7fd04f907700 5 osd.0 21 tick -3101> 2014-05-19 18:37:36.498807 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3100> 2014-05-19 18:37:36.498812 7fd04f907700 10 osd.0 21 do_waiters -- start -3099> 2014-05-19 18:37:36.498814 7fd04f907700 10 osd.0 21 do_waiters -- finish -3098> 2014-05-19 18:37:36.498843 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3097> 2014-05-19 18:37:36.569151 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:39.626800) v2 ==== 47+0+0 (1750282168 0 0) 0x51a6540 con 0x4cb9340 -3096> 2014-05-19 18:37:36.569179 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:39.626800) v2 ==== 47+0+0 (1750282168 0 0) 0x71221c0 con 0x4cb8dc0 -3095> 2014-05-19 18:37:36.569200 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:39.626800) v2 -- ?+0 0x6aa56c0 con 0x4cb9340 -3094> 2014-05-19 18:37:36.569278 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3093> 2014-05-19 18:37:36.569317 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:39.626800) v2 -- ?+0 0x2ee16c0 con 0x4cb8dc0 -3092> 2014-05-19 18:37:36.569354 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3091> 2014-05-19 18:37:36.694261 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:41.782888) v2 ==== 47+0+0 (66820870 0 0) 0x65421c0 con 0x5f5b760 -3090> 2014-05-19 18:37:36.694282 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:41.782888) v2 ==== 47+0+0 (66820870 0 0) 0x72de380 con 0x5b21600 -3089> 2014-05-19 18:37:36.694308 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:41.782888) v2 -- ?+0 0x51a6540 con 0x5f5b760 -3088> 2014-05-19 18:37:36.694342 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3087> 2014-05-19 18:37:36.694368 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:41.782888) v2 -- ?+0 0x71221c0 con 0x5b21600 -3086> 2014-05-19 18:37:36.694424 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -3085> 2014-05-19 18:37:36.777226 7fd04e905700 20 KeyValueStore::op_tp worker waiting -3084> 2014-05-19 18:37:36.777229 7fd043ebd700 20 OSD::op_tp worker waiting -3083> 2014-05-19 18:37:36.949377 7fd0396a8700 20 OSD::command_tp worker waiting -3082> 2014-05-19 18:37:37.343802 7fd0436bc700 20 OSD::op_tp worker waiting -3081> 2014-05-19 18:37:37.343825 7fd03c6ae700 20 OSD::op_tp worker waiting -3080> 2014-05-19 18:37:37.343847 7fd041eb9700 20 OSD::op_tp worker waiting -3079> 2014-05-19 18:37:37.343864 7fd03d6b0700 20 OSD::op_tp worker waiting -3078> 2014-05-19 18:37:37.343874 7fd03bead700 20 OSD::op_tp worker waiting -3077> 2014-05-19 18:37:37.343884 7fd03e6b2700 20 OSD::op_tp worker waiting -3076> 2014-05-19 18:37:37.343990 7fd03f6b4700 20 OSD::op_tp worker waiting -3075> 2014-05-19 18:37:37.358398 7fd03feb5700 20 OSD::op_tp worker waiting -3074> 2014-05-19 18:37:37.358480 7fd0446be700 20 OSD::op_tp worker waiting -3073> 2014-05-19 18:37:37.365425 7fd040eb7700 20 OSD::op_tp worker waiting -3072> 2014-05-19 18:37:37.370954 7fd0406b6700 20 OSD::op_tp worker waiting -3071> 2014-05-19 18:37:37.392384 7fd039ea9700 20 OSD::disk_tp worker waiting -3070> 2014-05-19 18:37:37.395959 7fd03aeab700 20 OSD::op_tp worker waiting -3069> 2014-05-19 18:37:37.395978 7fd03eeb3700 20 OSD::op_tp worker waiting -3068> 2014-05-19 18:37:37.414008 7fd042ebb700 20 OSD::op_tp worker waiting -3067> 2014-05-19 18:37:37.414034 7fd03deb1700 20 OSD::op_tp worker waiting -3066> 2014-05-19 18:37:37.422794 7fd04e104700 20 KeyValueStore::op_tp worker waiting -3065> 2014-05-19 18:37:37.422797 7fd03ceaf700 20 OSD::op_tp worker waiting -3064> 2014-05-19 18:37:37.422908 7fd0416b8700 20 OSD::op_tp worker waiting -3063> 2014-05-19 18:37:37.422966 7fd0426ba700 20 OSD::op_tp worker waiting -3062> 2014-05-19 18:37:37.422988 7fd03b6ac700 20 OSD::op_tp worker waiting -3061> 2014-05-19 18:37:37.498889 7fd04f907700 5 osd.0 21 tick -3060> 2014-05-19 18:37:37.498906 7fd04f907700 7 osd.0 21 do_mon_report -3059> 2014-05-19 18:37:37.498910 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -3058> 2014-05-19 18:37:37.498914 7fd04f907700 20 osd.0 21 send_pg_stats -3057> 2014-05-19 18:37:37.498922 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -3056> 2014-05-19 18:37:37.498926 7fd04f907700 10 osd.0 21 do_waiters -- start -3055> 2014-05-19 18:37:37.498928 7fd04f907700 10 osd.0 21 do_waiters -- finish -3054> 2014-05-19 18:37:37.498968 7fd03a6aa700 20 OSD::recovery_tp worker waiting -3053> 2014-05-19 18:37:37.619032 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:42.707702) v2 ==== 47+0+0 (890197148 0 0) 0x711e700 con 0x30946e0 -3052> 2014-05-19 18:37:37.619046 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 84 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:42.707702) v2 ==== 47+0+0 (890197148 0 0) 0x72e4fc0 con 0x3094160 -3051> 2014-05-19 18:37:37.619080 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.707702) v2 -- ?+0 0x72de380 con 0x30946e0 -3050> 2014-05-19 18:37:37.619112 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3049> 2014-05-19 18:37:37.619166 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.707702) v2 -- ?+0 0x65421c0 con 0x3094160 -3048> 2014-05-19 18:37:37.619214 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -3047> 2014-05-19 18:37:37.796160 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:37.795982) v2 ==== 47+0+0 (2564226699 0 0) 0x6112e00 con 0x4cb86e0 -3046> 2014-05-19 18:37:37.796190 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:37.795982) v2 ==== 47+0+0 (2564226699 0 0) 0x72e4c40 con 0x4cb8420 -3045> 2014-05-19 18:37:37.796209 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:37.795982) v2 -- ?+0 0x711e700 con 0x4cb86e0 -3044> 2014-05-19 18:37:37.796299 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3043> 2014-05-19 18:37:37.796367 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:37.795982) v2 -- ?+0 0x72e4fc0 con 0x4cb8420 -3042> 2014-05-19 18:37:37.796408 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3041> 2014-05-19 18:37:38.066462 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -3040> 2014-05-19 18:37:38.066481 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -3039> 2014-05-19 18:37:38.066489 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x60f61c0 con 0x3622580 -3038> 2014-05-19 18:37:38.066515 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x66f8e00 con 0x3622840 -3037> 2014-05-19 18:37:38.066552 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x6008e00 con 0x3622b00 -3036> 2014-05-19 18:37:38.066567 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x798fdc0 con 0x3622dc0 -3035> 2014-05-19 18:37:38.066582 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x7a5e1c0 con 0x30594a0 -3034> 2014-05-19 18:37:38.066608 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x8388a80 con 0x3622f20 -3033> 2014-05-19 18:37:38.066636 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x6504700 con 0x5e7ef20 -3032> 2014-05-19 18:37:38.066660 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x7a37180 con 0x5e7f1e0 -3031> 2014-05-19 18:37:38.066689 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x55f4380 con 0x5e7e2c0 -3030> 2014-05-19 18:37:38.066715 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x6d616c0 con 0x5d7bb80 -3029> 2014-05-19 18:37:38.066740 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x798e700 con 0x5e7ec60 -3028> 2014-05-19 18:37:38.066758 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x682fc00 con 0x531b760 -3027> 2014-05-19 18:37:38.066782 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x656c8c0 con 0x53bba20 -3026> 2014-05-19 18:37:38.066805 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x7735180 con 0x51cac60 -3025> 2014-05-19 18:37:38.066830 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x71cb340 con 0x5bd66e0 -3024> 2014-05-19 18:37:38.066857 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:38.066487) v2 -- ?+0 0x66f8700 con 0x5bd6840 -3023> 2014-05-19 18:37:38.066871 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x7931a40 con 0x3622b00 -3022> 2014-05-19 18:37:38.066919 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3021> 2014-05-19 18:37:38.066933 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x7122c40 con 0x3622580 -3020> 2014-05-19 18:37:38.066964 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3019> 2014-05-19 18:37:38.066975 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x799a1c0 con 0x3622840 -3018> 2014-05-19 18:37:38.066990 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -3017> 2014-05-19 18:37:38.067048 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x6378e00 con 0x5e7ef20 -3016> 2014-05-19 18:37:38.067085 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3015> 2014-05-19 18:37:38.067098 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x6009340 con 0x5e7e2c0 -3014> 2014-05-19 18:37:38.067142 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3013> 2014-05-19 18:37:38.067188 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x77461c0 con 0x30594a0 -3012> 2014-05-19 18:37:38.067222 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3011> 2014-05-19 18:37:38.067240 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x3510380 con 0x3622f20 -3010> 2014-05-19 18:37:38.067260 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -3009> 2014-05-19 18:37:38.067267 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x511ddc0 con 0x3622dc0 -3008> 2014-05-19 18:37:38.067295 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -3007> 2014-05-19 18:37:38.067301 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x6e2a380 con 0x5e7f1e0 -3006> 2014-05-19 18:37:38.067315 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -3005> 2014-05-19 18:37:38.067320 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x7999180 con 0x5d7bb80 -3004> 2014-05-19 18:37:38.067338 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -3003> 2014-05-19 18:37:38.067345 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x55c3880 con 0x5e7ec60 -3002> 2014-05-19 18:37:38.067360 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -3001> 2014-05-19 18:37:38.067365 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x73c28c0 con 0x531b760 -3000> 2014-05-19 18:37:38.067378 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2999> 2014-05-19 18:37:38.067398 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x5c041c0 con 0x53bba20 -2998> 2014-05-19 18:37:38.067415 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2997> 2014-05-19 18:37:38.067421 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x55181c0 con 0x51cac60 -2996> 2014-05-19 18:37:38.067435 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2995> 2014-05-19 18:37:38.067440 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x63d9340 con 0x5bd66e0 -2994> 2014-05-19 18:37:38.067453 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2993> 2014-05-19 18:37:38.067459 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 88 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:38.066487) v2 ==== 47+0+0 (3054385588 0 0) 0x7aa4e00 con 0x5bd6840 -2992> 2014-05-19 18:37:38.067472 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2991> 2014-05-19 18:37:38.499010 7fd04f907700 5 osd.0 21 tick -2990> 2014-05-19 18:37:38.499032 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2989> 2014-05-19 18:37:38.499036 7fd04f907700 10 osd.0 21 do_waiters -- start -2988> 2014-05-19 18:37:38.499039 7fd04f907700 10 osd.0 21 do_waiters -- finish -2987> 2014-05-19 18:37:38.499080 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2986> 2014-05-19 18:37:38.500046 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:41.557719) v2 ==== 47+0+0 (3605437825 0 0) 0x64a6700 con 0x61b0420 -2985> 2014-05-19 18:37:38.500093 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:41.557719) v2 -- ?+0 0x72e4c40 con 0x61b0420 -2984> 2014-05-19 18:37:38.500123 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2983> 2014-05-19 18:37:38.500140 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:41.557719) v2 ==== 47+0+0 (3605437825 0 0) 0x70ebc00 con 0x61ae160 -2982> 2014-05-19 18:37:38.500171 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:41.557719) v2 -- ?+0 0x6112e00 con 0x61ae160 -2981> 2014-05-19 18:37:38.500234 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2980> 2014-05-19 18:37:38.777313 7fd04e905700 20 KeyValueStore::op_tp worker waiting -2979> 2014-05-19 18:37:38.777321 7fd043ebd700 20 OSD::op_tp worker waiting -2978> 2014-05-19 18:37:38.869658 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:41.927341) v2 ==== 47+0+0 (3052322590 0 0) 0x5fc3880 con 0x4cb9340 -2977> 2014-05-19 18:37:38.869706 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:41.927341) v2 -- ?+0 0x64a6700 con 0x4cb9340 -2976> 2014-05-19 18:37:38.869779 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2975> 2014-05-19 18:37:38.869796 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:41.927341) v2 ==== 47+0+0 (3052322590 0 0) 0x7122700 con 0x4cb8dc0 -2974> 2014-05-19 18:37:38.869844 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:41.927341) v2 -- ?+0 0x70ebc00 con 0x4cb8dc0 -2973> 2014-05-19 18:37:38.869871 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2972> 2014-05-19 18:37:38.949470 7fd0396a8700 20 OSD::command_tp worker waiting -2971> 2014-05-19 18:37:39.343881 7fd0436bc700 20 OSD::op_tp worker waiting -2970> 2014-05-19 18:37:39.343903 7fd03c6ae700 20 OSD::op_tp worker waiting -2969> 2014-05-19 18:37:39.343925 7fd03bead700 20 OSD::op_tp worker waiting -2968> 2014-05-19 18:37:39.343943 7fd041eb9700 20 OSD::op_tp worker waiting -2967> 2014-05-19 18:37:39.343956 7fd03d6b0700 20 OSD::op_tp worker waiting -2966> 2014-05-19 18:37:39.343965 7fd03e6b2700 20 OSD::op_tp worker waiting -2965> 2014-05-19 18:37:39.344058 7fd03f6b4700 20 OSD::op_tp worker waiting -2964> 2014-05-19 18:37:39.358478 7fd03feb5700 20 OSD::op_tp worker waiting -2963> 2014-05-19 18:37:39.358571 7fd0446be700 20 OSD::op_tp worker waiting -2962> 2014-05-19 18:37:39.365521 7fd040eb7700 20 OSD::op_tp worker waiting -2961> 2014-05-19 18:37:39.371051 7fd0406b6700 20 OSD::op_tp worker waiting -2960> 2014-05-19 18:37:39.379375 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:44.468107) v2 ==== 47+0+0 (1452298739 0 0) 0x649e540 con 0x61b0160 -2959> 2014-05-19 18:37:39.379427 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:44.468107) v2 -- ?+0 0x5fc3880 con 0x61b0160 -2958> 2014-05-19 18:37:39.379459 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2957> 2014-05-19 18:37:39.379442 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:44.468107) v2 ==== 47+0+0 (1452298739 0 0) 0x5b28380 con 0x31e1600 -2956> 2014-05-19 18:37:39.379489 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:44.468107) v2 -- ?+0 0x7122700 con 0x31e1600 -2955> 2014-05-19 18:37:39.379531 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2954> 2014-05-19 18:37:39.392481 7fd039ea9700 20 OSD::disk_tp worker waiting -2953> 2014-05-19 18:37:39.396042 7fd03aeab700 20 OSD::op_tp worker waiting -2952> 2014-05-19 18:37:39.396063 7fd03eeb3700 20 OSD::op_tp worker waiting -2951> 2014-05-19 18:37:39.414089 7fd042ebb700 20 OSD::op_tp worker waiting -2950> 2014-05-19 18:37:39.414111 7fd03deb1700 20 OSD::op_tp worker waiting -2949> 2014-05-19 18:37:39.422907 7fd04e104700 20 KeyValueStore::op_tp worker waiting -2948> 2014-05-19 18:37:39.422931 7fd03ceaf700 20 OSD::op_tp worker waiting -2947> 2014-05-19 18:37:39.422980 7fd0416b8700 20 OSD::op_tp worker waiting -2946> 2014-05-19 18:37:39.423057 7fd03b6ac700 20 OSD::op_tp worker waiting -2945> 2014-05-19 18:37:39.423080 7fd0426ba700 20 OSD::op_tp worker waiting -2944> 2014-05-19 18:37:39.499125 7fd04f907700 5 osd.0 21 tick -2943> 2014-05-19 18:37:39.499146 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2942> 2014-05-19 18:37:39.499151 7fd04f907700 10 osd.0 21 do_waiters -- start -2941> 2014-05-19 18:37:39.499153 7fd04f907700 10 osd.0 21 do_waiters -- finish -2940> 2014-05-19 18:37:39.499194 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2939> 2014-05-19 18:37:39.831326 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:42.889015) v2 ==== 47+0+0 (3674850501 0 0) 0x77e8e00 con 0x61ae420 -2938> 2014-05-19 18:37:39.831347 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:42.889015) v2 ==== 47+0+0 (3674850501 0 0) 0x5519c00 con 0x618fe40 -2937> 2014-05-19 18:37:39.831375 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.889015) v2 -- ?+0 0x5b28380 con 0x61ae420 -2936> 2014-05-19 18:37:39.831453 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2935> 2014-05-19 18:37:39.831521 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.889015) v2 -- ?+0 0x649e540 con 0x618fe40 -2934> 2014-05-19 18:37:39.831566 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2933> 2014-05-19 18:37:39.907982 7fd0456c0700 10 monclient: tick -2932> 2014-05-19 18:37:39.907995 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:37:09.907994) -2931> 2014-05-19 18:37:39.908012 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:37:39.908012; renew after: 2014-05-19 18:38:19.906664) -- no -2930> 2014-05-19 18:37:39.982014 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:39.981785) v2 ==== 47+0+0 (2868067083 0 0) 0x6aa5c00 con 0x4cb82c0 -2929> 2014-05-19 18:37:39.982060 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:39.981785) v2 -- ?+0 0x5519c00 con 0x4cb82c0 -2928> 2014-05-19 18:37:39.982061 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:39.981785) v2 ==== 47+0+0 (2868067083 0 0) 0x2ee0fc0 con 0x4cb8160 -2927> 2014-05-19 18:37:39.982096 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2926> 2014-05-19 18:37:39.982123 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:39.981785) v2 -- ?+0 0x77e8e00 con 0x4cb8160 -2925> 2014-05-19 18:37:39.982229 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2924> 2014-05-19 18:37:40.499239 7fd04f907700 5 osd.0 21 tick -2923> 2014-05-19 18:37:40.499299 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2922> 2014-05-19 18:37:40.499296 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes -2921> 2014-05-19 18:37:40.499334 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -2920> 2014-05-19 18:37:40.499340 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 303.223 < min (86400 seconds) -2919> 2014-05-19 18:37:40.499367 7fd04f907700 20 osd.0 21 sched_scrub done -2918> 2014-05-19 18:37:40.499372 7fd04f907700 10 osd.0 21 do_waiters -- start -2917> 2014-05-19 18:37:40.499376 7fd04f907700 10 osd.0 21 do_waiters -- finish -2916> 2014-05-19 18:37:40.777382 7fd04e905700 20 KeyValueStore::op_tp worker waiting -2915> 2014-05-19 18:37:40.777400 7fd043ebd700 20 OSD::op_tp worker waiting -2914> 2014-05-19 18:37:40.800522 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:43.858218) v2 ==== 47+0+0 (646041485 0 0) 0x70ea700 con 0x61ae160 -2913> 2014-05-19 18:37:40.800572 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:43.858218) v2 -- ?+0 0x2ee0fc0 con 0x61ae160 -2912> 2014-05-19 18:37:40.800630 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2911> 2014-05-19 18:37:40.800644 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:43.858218) v2 ==== 47+0+0 (646041485 0 0) 0x3438700 con 0x61b0420 -2910> 2014-05-19 18:37:40.800681 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:43.858218) v2 -- ?+0 0x6aa5c00 con 0x61b0420 -2909> 2014-05-19 18:37:40.800707 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2908> 2014-05-19 18:37:40.949559 7fd0396a8700 20 OSD::command_tp worker waiting -2907> 2014-05-19 18:37:41.343961 7fd03c6ae700 20 OSD::op_tp worker waiting -2906> 2014-05-19 18:37:41.343984 7fd0436bc700 20 OSD::op_tp worker waiting -2905> 2014-05-19 18:37:41.344004 7fd03bead700 20 OSD::op_tp worker waiting -2904> 2014-05-19 18:37:41.344020 7fd03d6b0700 20 OSD::op_tp worker waiting -2903> 2014-05-19 18:37:41.344045 7fd041eb9700 20 OSD::op_tp worker waiting -2902> 2014-05-19 18:37:41.344058 7fd03e6b2700 20 OSD::op_tp worker waiting -2901> 2014-05-19 18:37:41.344119 7fd03f6b4700 20 OSD::op_tp worker waiting -2900> 2014-05-19 18:37:41.358551 7fd03feb5700 20 OSD::op_tp worker waiting -2899> 2014-05-19 18:37:41.358651 7fd0446be700 20 OSD::op_tp worker waiting -2898> 2014-05-19 18:37:41.365618 7fd040eb7700 20 OSD::op_tp worker waiting -2897> 2014-05-19 18:37:41.371147 7fd0406b6700 20 OSD::op_tp worker waiting -2896> 2014-05-19 18:37:41.392597 7fd039ea9700 20 OSD::disk_tp worker waiting -2895> 2014-05-19 18:37:41.396126 7fd03aeab700 20 OSD::op_tp worker waiting -2894> 2014-05-19 18:37:41.396146 7fd03eeb3700 20 OSD::op_tp worker waiting -2893> 2014-05-19 18:37:41.414171 7fd042ebb700 20 OSD::op_tp worker waiting -2892> 2014-05-19 18:37:41.414192 7fd03deb1700 20 OSD::op_tp worker waiting -2891> 2014-05-19 18:37:41.423009 7fd04e104700 20 KeyValueStore::op_tp worker waiting -2890> 2014-05-19 18:37:41.423011 7fd03ceaf700 20 OSD::op_tp worker waiting -2889> 2014-05-19 18:37:41.423042 7fd0416b8700 20 OSD::op_tp worker waiting -2888> 2014-05-19 18:37:41.423132 7fd03b6ac700 20 OSD::op_tp worker waiting -2887> 2014-05-19 18:37:41.423145 7fd0426ba700 20 OSD::op_tp worker waiting -2886> 2014-05-19 18:37:41.499458 7fd04f907700 5 osd.0 21 tick -2885> 2014-05-19 18:37:41.499506 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes -2884> 2014-05-19 18:37:41.499519 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -2883> 2014-05-19 18:37:41.499528 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2882> 2014-05-19 18:37:41.499522 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 304.224 < min (86400 seconds) -2881> 2014-05-19 18:37:41.499541 7fd04f907700 20 osd.0 21 sched_scrub done -2880> 2014-05-19 18:37:41.499544 7fd04f907700 10 osd.0 21 do_waiters -- start -2879> 2014-05-19 18:37:41.499546 7fd04f907700 10 osd.0 21 do_waiters -- finish -2878> 2014-05-19 18:37:41.679859 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:46.768676) v2 ==== 47+0+0 (3750500251 0 0) 0x7970e00 con 0x31e1600 -2877> 2014-05-19 18:37:41.679867 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:46.768676) v2 ==== 47+0+0 (3750500251 0 0) 0x649ea80 con 0x61b0160 -2876> 2014-05-19 18:37:41.679921 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:46.768676) v2 -- ?+0 0x70ea700 con 0x31e1600 -2875> 2014-05-19 18:37:41.679955 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2874> 2014-05-19 18:37:41.679981 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:46.768676) v2 -- ?+0 0x3438700 con 0x61b0160 -2873> 2014-05-19 18:37:41.680050 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2872> 2014-05-19 18:37:41.896721 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:41.896530) v2 ==== 47+0+0 (164575970 0 0) 0x70ea380 con 0x4cb86e0 -2871> 2014-05-19 18:37:41.896741 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:41.896530) v2 ==== 47+0+0 (164575970 0 0) 0x75b08c0 con 0x4cb8420 -2870> 2014-05-19 18:37:41.896771 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:41.896530) v2 -- ?+0 0x7970e00 con 0x4cb86e0 -2869> 2014-05-19 18:37:41.896850 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2868> 2014-05-19 18:37:41.896878 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:41.896530) v2 -- ?+0 0x649ea80 con 0x4cb8420 -2867> 2014-05-19 18:37:41.896915 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2866> 2014-05-19 18:37:42.167005 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -2865> 2014-05-19 18:37:42.167023 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -2864> 2014-05-19 18:37:42.167031 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x73c2000 con 0x3622580 -2863> 2014-05-19 18:37:42.167056 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x79988c0 con 0x3622840 -2862> 2014-05-19 18:37:42.167095 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x5f06000 con 0x3622b00 -2861> 2014-05-19 18:37:42.167110 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x83881c0 con 0x3622dc0 -2860> 2014-05-19 18:37:42.167145 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x7971dc0 con 0x30594a0 -2859> 2014-05-19 18:37:42.167174 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x55f48c0 con 0x3622f20 -2858> 2014-05-19 18:37:42.167200 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x7a5f340 con 0x5e7ef20 -2857> 2014-05-19 18:37:42.167225 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x6505500 con 0x5e7f1e0 -2856> 2014-05-19 18:37:42.167251 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x6b71a40 con 0x5e7e2c0 -2855> 2014-05-19 18:37:42.167277 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x60a0fc0 con 0x5d7bb80 -2854> 2014-05-19 18:37:42.167302 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x682fdc0 con 0x5e7ec60 -2853> 2014-05-19 18:37:42.167324 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x7a5f180 con 0x531b760 -2852> 2014-05-19 18:37:42.167349 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x656c1c0 con 0x53bba20 -2851> 2014-05-19 18:37:42.167373 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x7734000 con 0x51cac60 -2850> 2014-05-19 18:37:42.167398 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x60f6e00 con 0x5bd66e0 -2849> 2014-05-19 18:37:42.167424 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:42.167029) v2 -- ?+0 0x66f8000 con 0x5bd6840 -2848> 2014-05-19 18:37:42.167430 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x7122540 con 0x3622580 -2847> 2014-05-19 18:37:42.167466 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2846> 2014-05-19 18:37:42.167490 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x7930540 con 0x3622b00 -2845> 2014-05-19 18:37:42.167509 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2844> 2014-05-19 18:37:42.167527 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x55f5c00 con 0x3622840 -2843> 2014-05-19 18:37:42.167543 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2842> 2014-05-19 18:37:42.167563 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x7178c40 con 0x3622dc0 -2841> 2014-05-19 18:37:42.167597 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2840> 2014-05-19 18:37:42.167609 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x63781c0 con 0x5e7ef20 -2839> 2014-05-19 18:37:42.167643 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2838> 2014-05-19 18:37:42.167664 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x77468c0 con 0x30594a0 -2837> 2014-05-19 18:37:42.167694 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2836> 2014-05-19 18:37:42.167705 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x6009500 con 0x5e7e2c0 -2835> 2014-05-19 18:37:42.167721 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2834> 2014-05-19 18:37:42.167727 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x6e2a700 con 0x5e7f1e0 -2833> 2014-05-19 18:37:42.167776 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2832> 2014-05-19 18:37:42.167876 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x7998fc0 con 0x5d7bb80 -2831> 2014-05-19 18:37:42.167918 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2830> 2014-05-19 18:37:42.167929 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x860fa40 con 0x5e7ec60 -2829> 2014-05-19 18:37:42.167947 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2828> 2014-05-19 18:37:42.167956 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x73c2380 con 0x531b760 -2827> 2014-05-19 18:37:42.167984 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2826> 2014-05-19 18:37:42.167990 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x5c05340 con 0x53bba20 -2825> 2014-05-19 18:37:42.168005 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2824> 2014-05-19 18:37:42.168013 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x79e7340 con 0x51cac60 -2823> 2014-05-19 18:37:42.168028 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2822> 2014-05-19 18:37:42.168033 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x63d9a40 con 0x5bd66e0 -2821> 2014-05-19 18:37:42.168047 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2820> 2014-05-19 18:37:42.168052 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 89 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x7aa5340 con 0x5bd6840 -2819> 2014-05-19 18:37:42.168070 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2818> 2014-05-19 18:37:42.168078 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:42.167029) v2 ==== 47+0+0 (4176314696 0 0) 0x3511340 con 0x3622f20 -2817> 2014-05-19 18:37:42.168095 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2816> 2014-05-19 18:37:42.499620 7fd04f907700 5 osd.0 21 tick -2815> 2014-05-19 18:37:42.499637 7fd04f907700 7 osd.0 21 do_mon_report -2814> 2014-05-19 18:37:42.499641 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -2813> 2014-05-19 18:37:42.499645 7fd04f907700 20 osd.0 21 send_pg_stats -2812> 2014-05-19 18:37:42.499653 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2811> 2014-05-19 18:37:42.499657 7fd04f907700 10 osd.0 21 do_waiters -- start -2810> 2014-05-19 18:37:42.499659 7fd04f907700 10 osd.0 21 do_waiters -- finish -2809> 2014-05-19 18:37:42.499698 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2808> 2014-05-19 18:37:42.594515 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:47.683365) v2 ==== 47+0+0 (820448585 0 0) 0x653ea80 con 0x5f5b760 -2807> 2014-05-19 18:37:42.594568 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.683365) v2 -- ?+0 0x75b08c0 con 0x5f5b760 -2806> 2014-05-19 18:37:42.594509 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:47.683365) v2 ==== 47+0+0 (820448585 0 0) 0x60f6000 con 0x5b21600 -2805> 2014-05-19 18:37:42.594661 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2804> 2014-05-19 18:37:42.594689 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.683365) v2 -- ?+0 0x70ea380 con 0x5b21600 -2803> 2014-05-19 18:37:42.594720 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2802> 2014-05-19 18:37:42.777467 7fd04e905700 20 KeyValueStore::op_tp worker waiting -2801> 2014-05-19 18:37:42.777478 7fd043ebd700 20 OSD::op_tp worker waiting -2800> 2014-05-19 18:37:42.949647 7fd0396a8700 20 OSD::command_tp worker waiting -2799> 2014-05-19 18:37:42.970142 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:46.027865) v2 ==== 47+0+0 (3141336702 0 0) 0x5bea540 con 0x4cb9340 -2798> 2014-05-19 18:37:42.970191 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:46.027865) v2 -- ?+0 0x653ea80 con 0x4cb9340 -2797> 2014-05-19 18:37:42.970225 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2796> 2014-05-19 18:37:42.970266 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:46.027865) v2 ==== 47+0+0 (3141336702 0 0) 0x5519c00 con 0x4cb8dc0 -2795> 2014-05-19 18:37:42.970311 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:46.027865) v2 -- ?+0 0x60f6000 con 0x4cb8dc0 -2794> 2014-05-19 18:37:42.970331 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2793> 2014-05-19 18:37:43.344059 7fd03c6ae700 20 OSD::op_tp worker waiting -2792> 2014-05-19 18:37:43.344084 7fd0436bc700 20 OSD::op_tp worker waiting -2791> 2014-05-19 18:37:43.344104 7fd03bead700 20 OSD::op_tp worker waiting -2790> 2014-05-19 18:37:43.344128 7fd03d6b0700 20 OSD::op_tp worker waiting -2789> 2014-05-19 18:37:43.344144 7fd041eb9700 20 OSD::op_tp worker waiting -2788> 2014-05-19 18:37:43.344180 7fd03f6b4700 20 OSD::op_tp worker waiting -2787> 2014-05-19 18:37:43.344193 7fd03e6b2700 20 OSD::op_tp worker waiting -2786> 2014-05-19 18:37:43.358641 7fd03feb5700 20 OSD::op_tp worker waiting -2785> 2014-05-19 18:37:43.358726 7fd0446be700 20 OSD::op_tp worker waiting -2784> 2014-05-19 18:37:43.365715 7fd040eb7700 20 OSD::op_tp worker waiting -2783> 2014-05-19 18:37:43.371244 7fd0406b6700 20 OSD::op_tp worker waiting -2782> 2014-05-19 18:37:43.392692 7fd039ea9700 20 OSD::disk_tp worker waiting -2781> 2014-05-19 18:37:43.396206 7fd03aeab700 20 OSD::op_tp worker waiting -2780> 2014-05-19 18:37:43.396227 7fd03eeb3700 20 OSD::op_tp worker waiting -2779> 2014-05-19 18:37:43.414255 7fd042ebb700 20 OSD::op_tp worker waiting -2778> 2014-05-19 18:37:43.414275 7fd03deb1700 20 OSD::op_tp worker waiting -2777> 2014-05-19 18:37:43.423115 7fd04e104700 20 KeyValueStore::op_tp worker waiting -2776> 2014-05-19 18:37:43.423119 7fd03ceaf700 20 OSD::op_tp worker waiting -2775> 2014-05-19 18:37:43.423183 7fd0416b8700 20 OSD::op_tp worker waiting -2774> 2014-05-19 18:37:43.423217 7fd03b6ac700 20 OSD::op_tp worker waiting -2773> 2014-05-19 18:37:43.423241 7fd0426ba700 20 OSD::op_tp worker waiting -2772> 2014-05-19 18:37:43.499738 7fd04f907700 5 osd.0 21 tick -2771> 2014-05-19 18:37:43.499759 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2770> 2014-05-19 18:37:43.499763 7fd04f907700 10 osd.0 21 do_waiters -- start -2769> 2014-05-19 18:37:43.499766 7fd04f907700 10 osd.0 21 do_waiters -- finish -2768> 2014-05-19 18:37:43.499807 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2767> 2014-05-19 18:37:43.519321 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:48.608204) v2 ==== 47+0+0 (3500582731 0 0) 0x51a7dc0 con 0x3094160 -2766> 2014-05-19 18:37:43.519327 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 85 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:48.608204) v2 ==== 47+0+0 (3500582731 0 0) 0x62e0540 con 0x30946e0 -2765> 2014-05-19 18:37:43.519362 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:48.608204) v2 -- ?+0 0x5bea540 con 0x3094160 -2764> 2014-05-19 18:37:43.519444 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2763> 2014-05-19 18:37:43.519466 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:48.608204) v2 -- ?+0 0x5519c00 con 0x30946e0 -2762> 2014-05-19 18:37:43.519520 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2761> 2014-05-19 18:37:43.931756 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:46.989494) v2 ==== 47+0+0 (3736978915 0 0) 0x5519dc0 con 0x618fe40 -2760> 2014-05-19 18:37:43.931766 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:46.989494) v2 ==== 47+0+0 (3736978915 0 0) 0x63d8c40 con 0x61ae420 -2759> 2014-05-19 18:37:43.931809 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:46.989494) v2 -- ?+0 0x51a7dc0 con 0x618fe40 -2758> 2014-05-19 18:37:43.931842 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2757> 2014-05-19 18:37:43.931866 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:46.989494) v2 -- ?+0 0x62e0540 con 0x61ae420 -2756> 2014-05-19 18:37:43.931946 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2755> 2014-05-19 18:37:44.082587 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:44.082383) v2 ==== 47+0+0 (10792876 0 0) 0x2ee01c0 con 0x4cb8160 -2754> 2014-05-19 18:37:44.082607 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:44.082383) v2 ==== 47+0+0 (10792876 0 0) 0x70cd500 con 0x4cb82c0 -2753> 2014-05-19 18:37:44.082657 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:44.082383) v2 -- ?+0 0x63d8c40 con 0x4cb8160 -2752> 2014-05-19 18:37:44.082685 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2751> 2014-05-19 18:37:44.082706 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:44.082383) v2 -- ?+0 0x5519dc0 con 0x4cb82c0 -2750> 2014-05-19 18:37:44.082744 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2749> 2014-05-19 18:37:44.499827 7fd04f907700 5 osd.0 21 tick -2748> 2014-05-19 18:37:44.499875 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes -2747> 2014-05-19 18:37:44.499888 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -2746> 2014-05-19 18:37:44.499884 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2745> 2014-05-19 18:37:44.499891 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 307.224 < min (86400 seconds) -2744> 2014-05-19 18:37:44.499910 7fd04f907700 20 osd.0 21 sched_scrub done -2743> 2014-05-19 18:37:44.499913 7fd04f907700 10 osd.0 21 do_waiters -- start -2742> 2014-05-19 18:37:44.499915 7fd04f907700 10 osd.0 21 do_waiters -- finish -2741> 2014-05-19 18:37:44.777548 7fd04e905700 20 KeyValueStore::op_tp worker waiting -2740> 2014-05-19 18:37:44.777561 7fd043ebd700 20 OSD::op_tp worker waiting -2739> 2014-05-19 18:37:44.894928 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:49.983861) v2 ==== 47+0+0 (4286860485 0 0) 0x68aaa80 con 0x5f5b760 -2738> 2014-05-19 18:37:44.894936 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:49.983861) v2 ==== 47+0+0 (4286860485 0 0) 0x6378a80 con 0x5b21600 -2737> 2014-05-19 18:37:44.894978 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:49.983861) v2 -- ?+0 0x70cd500 con 0x5f5b760 -2736> 2014-05-19 18:37:44.895016 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2735> 2014-05-19 18:37:44.895042 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:49.983861) v2 -- ?+0 0x2ee01c0 con 0x5b21600 -2734> 2014-05-19 18:37:44.895104 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2733> 2014-05-19 18:37:44.900958 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:47.958716) v2 ==== 47+0+0 (3587420830 0 0) 0x70ebdc0 con 0x61ae160 -2732> 2014-05-19 18:37:44.900961 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:47.958716) v2 ==== 47+0+0 (3587420830 0 0) 0x65436c0 con 0x61b0420 -2731> 2014-05-19 18:37:44.901014 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.958716) v2 -- ?+0 0x6378a80 con 0x61ae160 -2730> 2014-05-19 18:37:44.901050 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2729> 2014-05-19 18:37:44.901094 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.958716) v2 -- ?+0 0x68aaa80 con 0x61b0420 -2728> 2014-05-19 18:37:44.901139 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2727> 2014-05-19 18:37:44.949745 7fd0396a8700 20 OSD::command_tp worker waiting -2726> 2014-05-19 18:37:45.344146 7fd03c6ae700 20 OSD::op_tp worker waiting -2725> 2014-05-19 18:37:45.344168 7fd0436bc700 20 OSD::op_tp worker waiting -2724> 2014-05-19 18:37:45.344192 7fd03bead700 20 OSD::op_tp worker waiting -2723> 2014-05-19 18:37:45.344204 7fd03d6b0700 20 OSD::op_tp worker waiting -2722> 2014-05-19 18:37:45.344224 7fd041eb9700 20 OSD::op_tp worker waiting -2721> 2014-05-19 18:37:45.344243 7fd03f6b4700 20 OSD::op_tp worker waiting -2720> 2014-05-19 18:37:45.344293 7fd03e6b2700 20 OSD::op_tp worker waiting -2719> 2014-05-19 18:37:45.358723 7fd03feb5700 20 OSD::op_tp worker waiting -2718> 2014-05-19 18:37:45.358800 7fd0446be700 20 OSD::op_tp worker waiting -2717> 2014-05-19 18:37:45.365815 7fd040eb7700 20 OSD::op_tp worker waiting -2716> 2014-05-19 18:37:45.371342 7fd0406b6700 20 OSD::op_tp worker waiting -2715> 2014-05-19 18:37:45.392787 7fd039ea9700 20 OSD::disk_tp worker waiting -2714> 2014-05-19 18:37:45.396290 7fd03aeab700 20 OSD::op_tp worker waiting -2713> 2014-05-19 18:37:45.396310 7fd03eeb3700 20 OSD::op_tp worker waiting -2712> 2014-05-19 18:37:45.414336 7fd042ebb700 20 OSD::op_tp worker waiting -2711> 2014-05-19 18:37:45.414358 7fd03deb1700 20 OSD::op_tp worker waiting -2710> 2014-05-19 18:37:45.423222 7fd04e104700 20 KeyValueStore::op_tp worker waiting -2709> 2014-05-19 18:37:45.423226 7fd03ceaf700 20 OSD::op_tp worker waiting -2708> 2014-05-19 18:37:45.423270 7fd0416b8700 20 OSD::op_tp worker waiting -2707> 2014-05-19 18:37:45.423295 7fd03b6ac700 20 OSD::op_tp worker waiting -2706> 2014-05-19 18:37:45.423316 7fd0426ba700 20 OSD::op_tp worker waiting -2705> 2014-05-19 18:37:45.499984 7fd04f907700 5 osd.0 21 tick -2704> 2014-05-19 18:37:45.500032 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes -2703> 2014-05-19 18:37:45.500045 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -2702> 2014-05-19 18:37:45.500053 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2701> 2014-05-19 18:37:45.500048 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 308.224 < min (86400 seconds) -2700> 2014-05-19 18:37:45.500067 7fd04f907700 20 osd.0 21 sched_scrub done -2699> 2014-05-19 18:37:45.500070 7fd04f907700 10 osd.0 21 do_waiters -- start -2698> 2014-05-19 18:37:45.500072 7fd04f907700 10 osd.0 21 do_waiters -- finish -2697> 2014-05-19 18:37:45.780324 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:50.869268) v2 ==== 47+0+0 (19998668 0 0) 0x79716c0 con 0x31e1600 -2696> 2014-05-19 18:37:45.780348 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:50.869268) v2 ==== 47+0+0 (19998668 0 0) 0x649e000 con 0x61b0160 -2695> 2014-05-19 18:37:45.780379 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:50.869268) v2 -- ?+0 0x70ebdc0 con 0x31e1600 -2694> 2014-05-19 18:37:45.780418 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2693> 2014-05-19 18:37:45.780444 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:50.869268) v2 -- ?+0 0x65436c0 con 0x61b0160 -2692> 2014-05-19 18:37:45.780473 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2691> 2014-05-19 18:37:45.819774 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:50.908744) v2 ==== 47+0+0 (1224116243 0 0) 0x612c8c0 con 0x3094160 -2690> 2014-05-19 18:37:45.819824 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:50.908744) v2 -- ?+0 0x649e000 con 0x3094160 -2689> 2014-05-19 18:37:45.819813 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 86 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:50.908744) v2 ==== 47+0+0 (1224116243 0 0) 0x2f4efc0 con 0x30946e0 -2688> 2014-05-19 18:37:45.819866 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2687> 2014-05-19 18:37:45.819890 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:50.908744) v2 -- ?+0 0x79716c0 con 0x30946e0 -2686> 2014-05-19 18:37:45.819926 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2685> 2014-05-19 18:37:45.997259 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:45.997103) v2 ==== 47+0+0 (377410949 0 0) 0x5e6c380 con 0x4cb86e0 -2684> 2014-05-19 18:37:45.997307 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:45.997103) v2 -- ?+0 0x2f4efc0 con 0x4cb86e0 -2683> 2014-05-19 18:37:45.997335 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2682> 2014-05-19 18:37:45.997328 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:45.997103) v2 ==== 47+0+0 (377410949 0 0) 0x75b0e00 con 0x4cb8420 -2681> 2014-05-19 18:37:45.997380 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:45.997103) v2 -- ?+0 0x612c8c0 con 0x4cb8420 -2680> 2014-05-19 18:37:45.997459 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2679> 2014-05-19 18:37:46.500139 7fd04f907700 5 osd.0 21 tick -2678> 2014-05-19 18:37:46.500187 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes -2677> 2014-05-19 18:37:46.500200 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -2676> 2014-05-19 18:37:46.500208 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2675> 2014-05-19 18:37:46.500203 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 309.224 < min (86400 seconds) -2674> 2014-05-19 18:37:46.500222 7fd04f907700 20 osd.0 21 sched_scrub done -2673> 2014-05-19 18:37:46.500225 7fd04f907700 10 osd.0 21 do_waiters -- start -2672> 2014-05-19 18:37:46.500227 7fd04f907700 10 osd.0 21 do_waiters -- finish -2671> 2014-05-19 18:37:46.777648 7fd04e905700 20 KeyValueStore::op_tp worker waiting -2670> 2014-05-19 18:37:46.777650 7fd043ebd700 20 OSD::op_tp worker waiting -2669> 2014-05-19 18:37:46.949838 7fd0396a8700 20 OSD::command_tp worker waiting -2668> 2014-05-19 18:37:47.070641 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:50.128425) v2 ==== 47+0+0 (3927241184 0 0) 0x5beaa80 con 0x4cb9340 -2667> 2014-05-19 18:37:47.070661 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:50.128425) v2 ==== 47+0+0 (3927241184 0 0) 0x5519500 con 0x4cb8dc0 -2666> 2014-05-19 18:37:47.070687 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:50.128425) v2 -- ?+0 0x75b0e00 con 0x4cb9340 -2665> 2014-05-19 18:37:47.070710 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2664> 2014-05-19 18:37:47.070732 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:50.128425) v2 -- ?+0 0x5e6c380 con 0x4cb8dc0 -2663> 2014-05-19 18:37:47.070768 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2662> 2014-05-19 18:37:47.344242 7fd03c6ae700 20 OSD::op_tp worker waiting -2661> 2014-05-19 18:37:47.344260 7fd03bead700 20 OSD::op_tp worker waiting -2660> 2014-05-19 18:37:47.344274 7fd0436bc700 20 OSD::op_tp worker waiting -2659> 2014-05-19 18:37:47.344292 7fd041eb9700 20 OSD::op_tp worker waiting -2658> 2014-05-19 18:37:47.344307 7fd03d6b0700 20 OSD::op_tp worker waiting -2657> 2014-05-19 18:37:47.344326 7fd03f6b4700 20 OSD::op_tp worker waiting -2656> 2014-05-19 18:37:47.344364 7fd03e6b2700 20 OSD::op_tp worker waiting -2655> 2014-05-19 18:37:47.358818 7fd03feb5700 20 OSD::op_tp worker waiting -2654> 2014-05-19 18:37:47.358870 7fd0446be700 20 OSD::op_tp worker waiting -2653> 2014-05-19 18:37:47.365912 7fd040eb7700 20 OSD::op_tp worker waiting -2652> 2014-05-19 18:37:47.371429 7fd0406b6700 20 OSD::op_tp worker waiting -2651> 2014-05-19 18:37:47.392883 7fd039ea9700 20 OSD::disk_tp worker waiting -2650> 2014-05-19 18:37:47.396391 7fd03aeab700 20 OSD::op_tp worker waiting -2649> 2014-05-19 18:37:47.396417 7fd03eeb3700 20 OSD::op_tp worker waiting -2648> 2014-05-19 18:37:47.414417 7fd042ebb700 20 OSD::op_tp worker waiting -2647> 2014-05-19 18:37:47.414435 7fd03deb1700 20 OSD::op_tp worker waiting -2646> 2014-05-19 18:37:47.423312 7fd04e104700 20 KeyValueStore::op_tp worker waiting -2645> 2014-05-19 18:37:47.423318 7fd03ceaf700 20 OSD::op_tp worker waiting -2644> 2014-05-19 18:37:47.423349 7fd0416b8700 20 OSD::op_tp worker waiting -2643> 2014-05-19 18:37:47.423371 7fd03b6ac700 20 OSD::op_tp worker waiting -2642> 2014-05-19 18:37:47.423393 7fd0426ba700 20 OSD::op_tp worker waiting -2641> 2014-05-19 18:37:47.467559 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -2640> 2014-05-19 18:37:47.467577 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -2639> 2014-05-19 18:37:47.467585 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x5c46700 con 0x3622580 -2638> 2014-05-19 18:37:47.467611 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x798fc00 con 0x3622840 -2637> 2014-05-19 18:37:47.467649 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x7999880 con 0x3622b00 -2636> 2014-05-19 18:37:47.467663 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x5f06540 con 0x3622dc0 -2635> 2014-05-19 18:37:47.467677 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x77461c0 con 0x30594a0 -2634> 2014-05-19 18:37:47.467705 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x6009340 con 0x3622f20 -2633> 2014-05-19 18:37:47.467741 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x6378e00 con 0x5e7ef20 -2632> 2014-05-19 18:37:47.467768 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x799a1c0 con 0x5e7f1e0 -2631> 2014-05-19 18:37:47.467800 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x7122c40 con 0x5e7e2c0 -2630> 2014-05-19 18:37:47.467829 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x7931a40 con 0x5d7bb80 -2629> 2014-05-19 18:37:47.467858 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x7aa5880 con 0x5e7ec60 -2628> 2014-05-19 18:37:47.467889 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x7734540 con 0x531b760 -2627> 2014-05-19 18:37:47.467908 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x5518700 con 0x53bba20 -2626> 2014-05-19 18:37:47.467923 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x5c05a40 con 0x51cac60 -2625> 2014-05-19 18:37:47.467947 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x73c3a40 con 0x5bd66e0 -2624> 2014-05-19 18:37:47.467976 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:47.467583) v2 -- ?+0 0x612ce00 con 0x5bd6840 -2623> 2014-05-19 18:37:47.467998 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x5f7b340 con 0x3622580 -2622> 2014-05-19 18:37:47.468048 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2621> 2014-05-19 18:37:47.468062 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x692d500 con 0x3622840 -2620> 2014-05-19 18:37:47.468116 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2619> 2014-05-19 18:37:47.468175 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x72e5c00 con 0x3622dc0 -2618> 2014-05-19 18:37:47.468212 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2617> 2014-05-19 18:37:47.468227 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x7733c00 con 0x5e7ef20 -2616> 2014-05-19 18:37:47.468244 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2615> 2014-05-19 18:37:47.468250 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x6b716c0 con 0x5e7e2c0 -2614> 2014-05-19 18:37:47.468271 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2613> 2014-05-19 18:37:47.468281 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x6d60700 con 0x30594a0 -2612> 2014-05-19 18:37:47.468311 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2611> 2014-05-19 18:37:47.468344 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x6009dc0 con 0x3622f20 -2610> 2014-05-19 18:37:47.468365 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2609> 2014-05-19 18:37:47.468381 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x6c4b180 con 0x3622b00 -2608> 2014-05-19 18:37:47.468410 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2607> 2014-05-19 18:37:47.468421 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x59bc1c0 con 0x5e7f1e0 -2606> 2014-05-19 18:37:47.468449 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2605> 2014-05-19 18:37:47.468459 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x77e96c0 con 0x5d7bb80 -2604> 2014-05-19 18:37:47.468474 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2603> 2014-05-19 18:37:47.468485 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x662b6c0 con 0x5e7ec60 -2602> 2014-05-19 18:37:47.468501 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2601> 2014-05-19 18:37:47.468509 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x66f8380 con 0x531b760 -2600> 2014-05-19 18:37:47.468524 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2599> 2014-05-19 18:37:47.468534 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x5c05500 con 0x53bba20 -2598> 2014-05-19 18:37:47.468550 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2597> 2014-05-19 18:37:47.468556 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x79e6c40 con 0x51cac60 -2596> 2014-05-19 18:37:47.468573 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2595> 2014-05-19 18:37:47.468602 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x63d9180 con 0x5bd66e0 -2594> 2014-05-19 18:37:47.468618 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2593> 2014-05-19 18:37:47.468624 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 90 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:47.467583) v2 ==== 47+0+0 (3021847391 0 0) 0x7aa4700 con 0x5bd6840 -2592> 2014-05-19 18:37:47.468639 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2591> 2014-05-19 18:37:47.500296 7fd04f907700 5 osd.0 21 tick -2590> 2014-05-19 18:37:47.500313 7fd04f907700 7 osd.0 21 do_mon_report -2589> 2014-05-19 18:37:47.500317 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -2588> 2014-05-19 18:37:47.500321 7fd04f907700 20 osd.0 21 send_pg_stats -2587> 2014-05-19 18:37:47.500336 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2586> 2014-05-19 18:37:47.500355 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes -2585> 2014-05-19 18:37:47.500367 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -2584> 2014-05-19 18:37:47.500370 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 310.224 < min (86400 seconds) -2583> 2014-05-19 18:37:47.500389 7fd04f907700 20 osd.0 21 sched_scrub done -2582> 2014-05-19 18:37:47.500392 7fd04f907700 10 osd.0 21 do_waiters -- start -2581> 2014-05-19 18:37:47.500394 7fd04f907700 10 osd.0 21 do_waiters -- finish -2580> 2014-05-19 18:37:48.500433 7fd04f907700 5 osd.0 21 tick -2579> 2014-05-19 18:37:48.500454 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2578> 2014-05-19 18:37:48.500459 7fd04f907700 10 osd.0 21 do_waiters -- start -2577> 2014-05-19 18:37:48.500461 7fd04f907700 10 osd.0 21 do_waiters -- finish -2576> 2014-05-19 18:37:48.500485 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2575> 2014-05-19 18:37:48.777755 7fd043ebd700 20 OSD::op_tp worker waiting -2574> 2014-05-19 18:37:48.777754 7fd04e905700 20 KeyValueStore::op_tp worker waiting -2573> 2014-05-19 18:37:48.949931 7fd0396a8700 20 OSD::command_tp worker waiting -2572> 2014-05-19 18:37:48.995256 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:54.084348) v2 ==== 47+0+0 (2236757280 0 0) 0x64ae700 con 0x5f5b760 -2571> 2014-05-19 18:37:48.995282 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:54.084348) v2 ==== 47+0+0 (2236757280 0 0) 0x788a000 con 0x5b21600 -2570> 2014-05-19 18:37:48.995308 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:54.084348) v2 -- ?+0 0x5beaa80 con 0x5f5b760 -2569> 2014-05-19 18:37:48.995343 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2568> 2014-05-19 18:37:48.995366 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:54.084348) v2 -- ?+0 0x5519500 con 0x5b21600 -2567> 2014-05-19 18:37:48.995455 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2566> 2014-05-19 18:37:49.001439 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:52.059226) v2 ==== 47+0+0 (3316772474 0 0) 0x75b08c0 con 0x61ae160 -2565> 2014-05-19 18:37:49.001466 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:52.059226) v2 ==== 47+0+0 (3316772474 0 0) 0x79e6540 con 0x61b0420 -2564> 2014-05-19 18:37:49.001486 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:52.059226) v2 -- ?+0 0x788a000 con 0x61ae160 -2563> 2014-05-19 18:37:49.001510 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2562> 2014-05-19 18:37:49.001532 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:52.059226) v2 -- ?+0 0x64ae700 con 0x61b0420 -2561> 2014-05-19 18:37:49.001569 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2560> 2014-05-19 18:37:49.232174 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:52.289965) v2 ==== 47+0+0 (1825030234 0 0) 0x662a1c0 con 0x61ae420 -2559> 2014-05-19 18:37:49.232196 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:52.289965) v2 ==== 47+0+0 (1825030234 0 0) 0x691a700 con 0x618fe40 -2558> 2014-05-19 18:37:49.232223 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:52.289965) v2 -- ?+0 0x75b08c0 con 0x61ae420 -2557> 2014-05-19 18:37:49.232255 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2556> 2014-05-19 18:37:49.232329 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:52.289965) v2 -- ?+0 0x79e6540 con 0x618fe40 -2555> 2014-05-19 18:37:49.232366 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2554> 2014-05-19 18:37:49.344333 7fd03c6ae700 20 OSD::op_tp worker waiting -2553> 2014-05-19 18:37:49.344361 7fd03bead700 20 OSD::op_tp worker waiting -2552> 2014-05-19 18:37:49.344380 7fd0436bc700 20 OSD::op_tp worker waiting -2551> 2014-05-19 18:37:49.344399 7fd041eb9700 20 OSD::op_tp worker waiting -2550> 2014-05-19 18:37:49.344429 7fd03e6b2700 20 OSD::op_tp worker waiting -2549> 2014-05-19 18:37:49.344446 7fd03f6b4700 20 OSD::op_tp worker waiting -2548> 2014-05-19 18:37:49.344471 7fd03d6b0700 20 OSD::op_tp worker waiting -2547> 2014-05-19 18:37:49.358909 7fd03feb5700 20 OSD::op_tp worker waiting -2546> 2014-05-19 18:37:49.358945 7fd0446be700 20 OSD::op_tp worker waiting -2545> 2014-05-19 18:37:49.366009 7fd040eb7700 20 OSD::op_tp worker waiting -2544> 2014-05-19 18:37:49.371510 7fd0406b6700 20 OSD::op_tp worker waiting -2543> 2014-05-19 18:37:49.383188 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:49.383010) v2 ==== 47+0+0 (1835641510 0 0) 0x656c700 con 0x4cb8160 -2542> 2014-05-19 18:37:49.383242 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:49.383010) v2 -- ?+0 0x662a1c0 con 0x4cb8160 -2541> 2014-05-19 18:37:49.383274 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2540> 2014-05-19 18:37:49.383193 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:49.383010) v2 ==== 47+0+0 (1835641510 0 0) 0x73c0000 con 0x4cb82c0 -2539> 2014-05-19 18:37:49.383323 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:49.383010) v2 -- ?+0 0x691a700 con 0x4cb82c0 -2538> 2014-05-19 18:37:49.383362 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2537> 2014-05-19 18:37:49.392982 7fd039ea9700 20 OSD::disk_tp worker waiting -2536> 2014-05-19 18:37:49.396490 7fd03aeab700 20 OSD::op_tp worker waiting -2535> 2014-05-19 18:37:49.396514 7fd03eeb3700 20 OSD::op_tp worker waiting -2534> 2014-05-19 18:37:49.414498 7fd042ebb700 20 OSD::op_tp worker waiting -2533> 2014-05-19 18:37:49.414521 7fd03deb1700 20 OSD::op_tp worker waiting -2532> 2014-05-19 18:37:49.423398 7fd04e104700 20 KeyValueStore::op_tp worker waiting -2531> 2014-05-19 18:37:49.423403 7fd03ceaf700 20 OSD::op_tp worker waiting -2530> 2014-05-19 18:37:49.423445 7fd0416b8700 20 OSD::op_tp worker waiting -2529> 2014-05-19 18:37:49.423456 7fd0426ba700 20 OSD::op_tp worker waiting -2528> 2014-05-19 18:37:49.423467 7fd03b6ac700 20 OSD::op_tp worker waiting -2527> 2014-05-19 18:37:49.500543 7fd04f907700 5 osd.0 21 tick -2526> 2014-05-19 18:37:49.500565 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2525> 2014-05-19 18:37:49.500569 7fd04f907700 10 osd.0 21 do_waiters -- start -2524> 2014-05-19 18:37:49.500572 7fd04f907700 10 osd.0 21 do_waiters -- finish -2523> 2014-05-19 18:37:49.500613 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2522> 2014-05-19 18:37:49.880783 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:54.969897) v2 ==== 47+0+0 (1375881351 0 0) 0x60a0700 con 0x31e1600 -2521> 2014-05-19 18:37:49.880788 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:54.969897) v2 ==== 47+0+0 (1375881351 0 0) 0x725d880 con 0x61b0160 -2520> 2014-05-19 18:37:49.880828 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:54.969897) v2 -- ?+0 0x656c700 con 0x31e1600 -2519> 2014-05-19 18:37:49.880864 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2518> 2014-05-19 18:37:49.880946 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:54.969897) v2 -- ?+0 0x73c0000 con 0x61b0160 -2517> 2014-05-19 18:37:49.880982 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2516> 2014-05-19 18:37:49.908099 7fd0456c0700 10 monclient: tick -2515> 2014-05-19 18:37:49.908111 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:37:19.908110) -2514> 2014-05-19 18:37:49.908128 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:37:49.908128; renew after: 2014-05-19 18:38:19.906664) -- no -2513> 2014-05-19 18:37:49.920156 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:55.009272) v2 ==== 47+0+0 (3132782672 0 0) 0x799bdc0 con 0x30946e0 -2512> 2014-05-19 18:37:49.920177 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 87 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:55.009272) v2 ==== 47+0+0 (3132782672 0 0) 0x612dc00 con 0x3094160 -2511> 2014-05-19 18:37:49.920202 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:55.009272) v2 -- ?+0 0x60a0700 con 0x30946e0 -2510> 2014-05-19 18:37:49.920270 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2509> 2014-05-19 18:37:49.920327 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:55.009272) v2 -- ?+0 0x725d880 con 0x3094160 -2508> 2014-05-19 18:37:49.920368 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2507> 2014-05-19 18:37:50.500676 7fd04f907700 5 osd.0 21 tick -2506> 2014-05-19 18:37:50.500724 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes -2505> 2014-05-19 18:37:50.500737 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -2504> 2014-05-19 18:37:50.500733 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2503> 2014-05-19 18:37:50.500740 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 313.225 < min (86400 seconds) -2502> 2014-05-19 18:37:50.500759 7fd04f907700 20 osd.0 21 sched_scrub done -2501> 2014-05-19 18:37:50.500761 7fd04f907700 10 osd.0 21 do_waiters -- start -2500> 2014-05-19 18:37:50.500763 7fd04f907700 10 osd.0 21 do_waiters -- finish -2499> 2014-05-19 18:37:50.777837 7fd04e905700 20 KeyValueStore::op_tp worker waiting -2498> 2014-05-19 18:37:50.777852 7fd043ebd700 20 OSD::op_tp worker waiting -2497> 2014-05-19 18:37:50.950025 7fd0396a8700 20 OSD::command_tp worker waiting -2496> 2014-05-19 18:37:51.297876 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:51.297676) v2 ==== 47+0+0 (824640237 0 0) 0x71236c0 con 0x4cb86e0 -2495> 2014-05-19 18:37:51.297890 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:51.297676) v2 ==== 47+0+0 (824640237 0 0) 0x5fc3180 con 0x4cb8420 -2494> 2014-05-19 18:37:51.297927 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.297676) v2 -- ?+0 0x799bdc0 con 0x4cb86e0 -2493> 2014-05-19 18:37:51.297959 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2492> 2014-05-19 18:37:51.297982 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.297676) v2 -- ?+0 0x612dc00 con 0x4cb8420 -2491> 2014-05-19 18:37:51.298074 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2490> 2014-05-19 18:37:51.344429 7fd03c6ae700 20 OSD::op_tp worker waiting -2489> 2014-05-19 18:37:51.344451 7fd0436bc700 20 OSD::op_tp worker waiting -2488> 2014-05-19 18:37:51.344474 7fd03bead700 20 OSD::op_tp worker waiting -2487> 2014-05-19 18:37:51.344492 7fd041eb9700 20 OSD::op_tp worker waiting -2486> 2014-05-19 18:37:51.344522 7fd03e6b2700 20 OSD::op_tp worker waiting -2485> 2014-05-19 18:37:51.344547 7fd03d6b0700 20 OSD::op_tp worker waiting -2484> 2014-05-19 18:37:51.344566 7fd03f6b4700 20 OSD::op_tp worker waiting -2483> 2014-05-19 18:37:51.359012 7fd0446be700 20 OSD::op_tp worker waiting -2482> 2014-05-19 18:37:51.359040 7fd03feb5700 20 OSD::op_tp worker waiting -2481> 2014-05-19 18:37:51.366107 7fd040eb7700 20 OSD::op_tp worker waiting -2480> 2014-05-19 18:37:51.371591 7fd0406b6700 20 OSD::op_tp worker waiting -2479> 2014-05-19 18:37:51.393079 7fd039ea9700 20 OSD::disk_tp worker waiting -2478> 2014-05-19 18:37:51.396590 7fd03aeab700 20 OSD::op_tp worker waiting -2477> 2014-05-19 18:37:51.396614 7fd03eeb3700 20 OSD::op_tp worker waiting -2476> 2014-05-19 18:37:51.414597 7fd042ebb700 20 OSD::op_tp worker waiting -2475> 2014-05-19 18:37:51.414619 7fd03deb1700 20 OSD::op_tp worker waiting -2474> 2014-05-19 18:37:51.423496 7fd04e104700 20 KeyValueStore::op_tp worker waiting -2473> 2014-05-19 18:37:51.423499 7fd03ceaf700 20 OSD::op_tp worker waiting -2472> 2014-05-19 18:37:51.423542 7fd0426ba700 20 OSD::op_tp worker waiting -2471> 2014-05-19 18:37:51.423563 7fd03b6ac700 20 OSD::op_tp worker waiting -2470> 2014-05-19 18:37:51.423586 7fd0416b8700 20 OSD::op_tp worker waiting -2469> 2014-05-19 18:37:51.500833 7fd04f907700 5 osd.0 21 tick -2468> 2014-05-19 18:37:51.500854 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2467> 2014-05-19 18:37:51.500858 7fd04f907700 10 osd.0 21 do_waiters -- start -2466> 2014-05-19 18:37:51.500860 7fd04f907700 10 osd.0 21 do_waiters -- finish -2465> 2014-05-19 18:37:51.500889 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2464> 2014-05-19 18:37:51.568137 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -2463> 2014-05-19 18:37:51.568155 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -2462> 2014-05-19 18:37:51.568163 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x6e2ba40 con 0x3622580 -2461> 2014-05-19 18:37:51.568188 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x3511180 con 0x3622840 -2460> 2014-05-19 18:37:51.568225 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x7747180 con 0x3622b00 -2459> 2014-05-19 18:37:51.568239 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x7998c40 con 0x3622dc0 -2458> 2014-05-19 18:37:51.568253 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x6009880 con 0x30594a0 -2457> 2014-05-19 18:37:51.568280 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x7971500 con 0x3622f20 -2456> 2014-05-19 18:37:51.568359 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x6504e00 con 0x5e7ef20 -2455> 2014-05-19 18:37:51.568389 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x6d61180 con 0x5e7f1e0 -2454> 2014-05-19 18:37:51.568417 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x7a37880 con 0x5e7e2c0 -2453> 2014-05-19 18:37:51.568445 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x74a76c0 con 0x5d7bb80 -2452> 2014-05-19 18:37:51.568477 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x7aa41c0 con 0x5e7ec60 -2451> 2014-05-19 18:37:51.568504 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x7734fc0 con 0x531b760 -2450> 2014-05-19 18:37:51.568533 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x5519880 con 0x53bba20 -2449> 2014-05-19 18:37:51.568554 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x860f6c0 con 0x3622b00 -2448> 2014-05-19 18:37:51.568572 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x5c04000 con 0x51cac60 -2447> 2014-05-19 18:37:51.568621 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x73c2540 con 0x5bd66e0 -2446> 2014-05-19 18:37:51.568650 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:51.568161) v2 -- ?+0 0x612ca80 con 0x5bd6840 -2445> 2014-05-19 18:37:51.568718 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2444> 2014-05-19 18:37:51.568746 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x5f7ac40 con 0x3622580 -2443> 2014-05-19 18:37:51.568770 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2442> 2014-05-19 18:37:51.568781 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x692cc40 con 0x3622840 -2441> 2014-05-19 18:37:51.568799 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2440> 2014-05-19 18:37:51.568805 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x72e5dc0 con 0x3622dc0 -2439> 2014-05-19 18:37:51.568824 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2438> 2014-05-19 18:37:51.568833 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x59bd500 con 0x5e7f1e0 -2437> 2014-05-19 18:37:51.568847 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2436> 2014-05-19 18:37:51.568858 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x6b70540 con 0x5e7e2c0 -2435> 2014-05-19 18:37:51.568875 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2434> 2014-05-19 18:37:51.568888 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x6d616c0 con 0x30594a0 -2433> 2014-05-19 18:37:51.568915 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2432> 2014-05-19 18:37:51.568926 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x77336c0 con 0x5e7ef20 -2431> 2014-05-19 18:37:51.568952 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2430> 2014-05-19 18:37:51.568962 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x77e8e00 con 0x5d7bb80 -2429> 2014-05-19 18:37:51.568977 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2428> 2014-05-19 18:37:51.568983 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x66f9500 con 0x531b760 -2427> 2014-05-19 18:37:51.569019 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2426> 2014-05-19 18:37:51.569028 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x6c4d500 con 0x53bba20 -2425> 2014-05-19 18:37:51.569044 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2424> 2014-05-19 18:37:51.569051 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x63d9dc0 con 0x5bd66e0 -2423> 2014-05-19 18:37:51.569065 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2422> 2014-05-19 18:37:51.569073 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x6008380 con 0x3622f20 -2421> 2014-05-19 18:37:51.569087 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2420> 2014-05-19 18:37:51.569093 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x5497a40 con 0x51cac60 -2419> 2014-05-19 18:37:51.569107 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2418> 2014-05-19 18:37:51.569116 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 91 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x7aa5180 con 0x5bd6840 -2417> 2014-05-19 18:37:51.569131 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2416> 2014-05-19 18:37:51.569136 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:51.568161) v2 ==== 47+0+0 (1745838236 0 0) 0x662b500 con 0x5e7ec60 -2415> 2014-05-19 18:37:51.569153 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2414> 2014-05-19 18:37:52.371198 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:55.429037) v2 ==== 47+0+0 (487172683 0 0) 0x5beb340 con 0x4cb9340 -2413> 2014-05-19 18:37:52.371216 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:55.429037) v2 ==== 47+0+0 (487172683 0 0) 0x669e540 con 0x4cb8dc0 -2412> 2014-05-19 18:37:52.371248 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:55.429037) v2 -- ?+0 0x5fc3180 con 0x4cb9340 -2411> 2014-05-19 18:37:52.371277 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2410> 2014-05-19 18:37:52.371340 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:55.429037) v2 -- ?+0 0x71236c0 con 0x4cb8dc0 -2409> 2014-05-19 18:37:52.371385 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2408> 2014-05-19 18:37:52.500940 7fd04f907700 5 osd.0 21 tick -2407> 2014-05-19 18:37:52.500957 7fd04f907700 7 osd.0 21 do_mon_report -2406> 2014-05-19 18:37:52.500961 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -2405> 2014-05-19 18:37:52.500964 7fd04f907700 20 osd.0 21 send_pg_stats -2404> 2014-05-19 18:37:52.500972 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2403> 2014-05-19 18:37:52.500975 7fd04f907700 10 osd.0 21 do_waiters -- start -2402> 2014-05-19 18:37:52.500978 7fd04f907700 10 osd.0 21 do_waiters -- finish -2401> 2014-05-19 18:37:52.501009 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2400> 2014-05-19 18:37:52.777933 7fd04e905700 20 KeyValueStore::op_tp worker waiting -2399> 2014-05-19 18:37:52.777937 7fd043ebd700 20 OSD::op_tp worker waiting -2398> 2014-05-19 18:37:52.950107 7fd0396a8700 20 OSD::command_tp worker waiting -2397> 2014-05-19 18:37:53.095575 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:58.184826) v2 ==== 47+0+0 (4105154841 0 0) 0x2f4fdc0 con 0x5f5b760 -2396> 2014-05-19 18:37:53.095578 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:58.184826) v2 ==== 47+0+0 (4105154841 0 0) 0x788a700 con 0x5b21600 -2395> 2014-05-19 18:37:53.095631 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:58.184826) v2 -- ?+0 0x5beb340 con 0x5f5b760 -2394> 2014-05-19 18:37:53.095673 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2393> 2014-05-19 18:37:53.095751 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:58.184826) v2 -- ?+0 0x669e540 con 0x5b21600 -2392> 2014-05-19 18:37:53.095796 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2391> 2014-05-19 18:37:53.332571 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:56.390449) v2 ==== 47+0+0 (2004444859 0 0) 0x662bdc0 con 0x61ae420 -2390> 2014-05-19 18:37:53.332628 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:56.390449) v2 ==== 47+0+0 (2004444859 0 0) 0x691bdc0 con 0x618fe40 -2389> 2014-05-19 18:37:53.332636 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.390449) v2 -- ?+0 0x788a700 con 0x61ae420 -2388> 2014-05-19 18:37:53.332706 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2387> 2014-05-19 18:37:53.332729 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.390449) v2 -- ?+0 0x2f4fdc0 con 0x618fe40 -2386> 2014-05-19 18:37:53.332755 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2385> 2014-05-19 18:37:53.344529 7fd03c6ae700 20 OSD::op_tp worker waiting -2384> 2014-05-19 18:37:53.344552 7fd0436bc700 20 OSD::op_tp worker waiting -2383> 2014-05-19 18:37:53.344573 7fd03bead700 20 OSD::op_tp worker waiting -2382> 2014-05-19 18:37:53.344603 7fd03f6b4700 20 OSD::op_tp worker waiting -2381> 2014-05-19 18:37:53.344620 7fd041eb9700 20 OSD::op_tp worker waiting -2380> 2014-05-19 18:37:53.344643 7fd03d6b0700 20 OSD::op_tp worker waiting -2379> 2014-05-19 18:37:53.344697 7fd03e6b2700 20 OSD::op_tp worker waiting -2378> 2014-05-19 18:37:53.359112 7fd0446be700 20 OSD::op_tp worker waiting -2377> 2014-05-19 18:37:53.359138 7fd03feb5700 20 OSD::op_tp worker waiting -2376> 2014-05-19 18:37:53.366203 7fd040eb7700 20 OSD::op_tp worker waiting -2375> 2014-05-19 18:37:53.371672 7fd0406b6700 20 OSD::op_tp worker waiting -2374> 2014-05-19 18:37:53.393175 7fd039ea9700 20 OSD::disk_tp worker waiting -2373> 2014-05-19 18:37:53.396690 7fd03aeab700 20 OSD::op_tp worker waiting -2372> 2014-05-19 18:37:53.396719 7fd03eeb3700 20 OSD::op_tp worker waiting -2371> 2014-05-19 18:37:53.414696 7fd042ebb700 20 OSD::op_tp worker waiting -2370> 2014-05-19 18:37:53.414720 7fd03deb1700 20 OSD::op_tp worker waiting -2369> 2014-05-19 18:37:53.423590 7fd04e104700 20 KeyValueStore::op_tp worker waiting -2368> 2014-05-19 18:37:53.423594 7fd03ceaf700 20 OSD::op_tp worker waiting -2367> 2014-05-19 18:37:53.423622 7fd0426ba700 20 OSD::op_tp worker waiting -2366> 2014-05-19 18:37:53.423639 7fd03b6ac700 20 OSD::op_tp worker waiting -2365> 2014-05-19 18:37:53.423694 7fd0416b8700 20 OSD::op_tp worker waiting -2364> 2014-05-19 18:37:53.483924 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:53.483614) v2 ==== 47+0+0 (321225661 0 0) 0x73c16c0 con 0x4cb82c0 -2363> 2014-05-19 18:37:53.483932 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:53.483614) v2 ==== 47+0+0 (321225661 0 0) 0x656cc40 con 0x4cb8160 -2362> 2014-05-19 18:37:53.483963 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.483614) v2 -- ?+0 0x691bdc0 con 0x4cb82c0 -2361> 2014-05-19 18:37:53.483997 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2360> 2014-05-19 18:37:53.484021 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.483614) v2 -- ?+0 0x662bdc0 con 0x4cb8160 -2359> 2014-05-19 18:37:53.484057 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2358> 2014-05-19 18:37:53.501048 7fd04f907700 5 osd.0 21 tick -2357> 2014-05-19 18:37:53.501096 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes -2356> 2014-05-19 18:37:53.501109 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -2355> 2014-05-19 18:37:53.501117 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2354> 2014-05-19 18:37:53.501112 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 316.225 < min (86400 seconds) -2353> 2014-05-19 18:37:53.501131 7fd04f907700 20 osd.0 21 sched_scrub done -2352> 2014-05-19 18:37:53.501134 7fd04f907700 10 osd.0 21 do_waiters -- start -2351> 2014-05-19 18:37:53.501135 7fd04f907700 10 osd.0 21 do_waiters -- finish -2350> 2014-05-19 18:37:53.868779 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -2349> 2014-05-19 18:37:53.868797 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -2348> 2014-05-19 18:37:53.868804 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x7998700 con 0x3622580 -2347> 2014-05-19 18:37:53.868830 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x5f07340 con 0x3622840 -2346> 2014-05-19 18:37:53.868868 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x7970540 con 0x3622b00 -2345> 2014-05-19 18:37:53.868883 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x682e540 con 0x3622dc0 -2344> 2014-05-19 18:37:53.868911 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x6d60700 con 0x30594a0 -2343> 2014-05-19 18:37:53.868937 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x6b716c0 con 0x3622f20 -2342> 2014-05-19 18:37:53.869033 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x7733c00 con 0x5e7ef20 -2341> 2014-05-19 18:37:53.869062 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x72e5c00 con 0x5e7f1e0 -2340> 2014-05-19 18:37:53.869086 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x692d500 con 0x5e7e2c0 -2339> 2014-05-19 18:37:53.869109 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x5f7b340 con 0x5d7bb80 -2338> 2014-05-19 18:37:53.869131 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x3511340 con 0x5e7ec60 -2337> 2014-05-19 18:37:53.869153 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x7aa5340 con 0x531b760 -2336> 2014-05-19 18:37:53.869178 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x63d9a40 con 0x53bba20 -2335> 2014-05-19 18:37:53.869175 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x5f7a8c0 con 0x3622580 -2334> 2014-05-19 18:37:53.869203 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x79e7340 con 0x51cac60 -2333> 2014-05-19 18:37:53.869228 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x5c05340 con 0x5bd66e0 -2332> 2014-05-19 18:37:53.869254 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:53.868803) v2 -- ?+0 0x73c2380 con 0x5bd6840 -2331> 2014-05-19 18:37:53.869332 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2330> 2014-05-19 18:37:53.869356 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x72e4c40 con 0x3622dc0 -2329> 2014-05-19 18:37:53.869386 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2328> 2014-05-19 18:37:53.869397 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x711e8c0 con 0x3622840 -2327> 2014-05-19 18:37:53.869434 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2326> 2014-05-19 18:37:53.869446 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x5ccc1c0 con 0x3622b00 -2325> 2014-05-19 18:37:53.869466 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2324> 2014-05-19 18:37:53.869478 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x77321c0 con 0x5e7ef20 -2323> 2014-05-19 18:37:53.869497 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2322> 2014-05-19 18:37:53.869510 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x59bcc40 con 0x5e7f1e0 -2321> 2014-05-19 18:37:53.869555 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2320> 2014-05-19 18:37:53.869577 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x5f32380 con 0x30594a0 -2319> 2014-05-19 18:37:53.869617 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2318> 2014-05-19 18:37:53.869631 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x6009c00 con 0x3622f20 -2317> 2014-05-19 18:37:53.869646 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2316> 2014-05-19 18:37:53.869652 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x77476c0 con 0x5e7e2c0 -2315> 2014-05-19 18:37:53.869683 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2314> 2014-05-19 18:37:53.869690 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x77e8380 con 0x5d7bb80 -2313> 2014-05-19 18:37:53.869709 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2312> 2014-05-19 18:37:53.869720 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x66f88c0 con 0x531b760 -2311> 2014-05-19 18:37:53.869735 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2310> 2014-05-19 18:37:53.869740 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x6c4c700 con 0x53bba20 -2309> 2014-05-19 18:37:53.869754 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2308> 2014-05-19 18:37:53.869760 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x5496000 con 0x51cac60 -2307> 2014-05-19 18:37:53.869774 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2306> 2014-05-19 18:37:53.869779 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x3524380 con 0x5bd66e0 -2305> 2014-05-19 18:37:53.869793 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2304> 2014-05-19 18:37:53.869800 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 92 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x5cbcfc0 con 0x5bd6840 -2303> 2014-05-19 18:37:53.869814 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2302> 2014-05-19 18:37:53.869822 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:53.868803) v2 ==== 47+0+0 (2529466884 0 0) 0x662b180 con 0x5e7ec60 -2301> 2014-05-19 18:37:53.869837 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2300> 2014-05-19 18:37:54.020473 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:59.109751) v2 ==== 47+0+0 (3836447835 0 0) 0x799a380 con 0x30946e0 -2299> 2014-05-19 18:37:54.020518 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:59.109751) v2 -- ?+0 0x656cc40 con 0x30946e0 -2298> 2014-05-19 18:37:54.020554 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2297> 2014-05-19 18:37:54.020467 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 88 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:59.109751) v2 ==== 47+0+0 (3836447835 0 0) 0x55f56c0 con 0x3094160 -2296> 2014-05-19 18:37:54.020745 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:59.109751) v2 -- ?+0 0x73c16c0 con 0x3094160 -2295> 2014-05-19 18:37:54.020787 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2294> 2014-05-19 18:37:54.301830 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:57.359679) v2 ==== 47+0+0 (922144034 0 0) 0x7122a80 con 0x61b0420 -2293> 2014-05-19 18:37:54.301869 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:57.359679) v2 -- ?+0 0x55f56c0 con 0x61b0420 -2292> 2014-05-19 18:37:54.301878 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:57.359679) v2 ==== 47+0+0 (922144034 0 0) 0x51a6540 con 0x61ae160 -2291> 2014-05-19 18:37:54.301900 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2290> 2014-05-19 18:37:54.301997 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:57.359679) v2 -- ?+0 0x799a380 con 0x61ae160 -2289> 2014-05-19 18:37:54.302030 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2288> 2014-05-19 18:37:54.501219 7fd04f907700 5 osd.0 21 tick -2287> 2014-05-19 18:37:54.501242 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2286> 2014-05-19 18:37:54.501248 7fd04f907700 10 osd.0 21 do_waiters -- start -2285> 2014-05-19 18:37:54.501253 7fd04f907700 10 osd.0 21 do_waiters -- finish -2284> 2014-05-19 18:37:54.501280 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2283> 2014-05-19 18:37:54.778022 7fd043ebd700 20 OSD::op_tp worker waiting -2282> 2014-05-19 18:37:54.778022 7fd04e905700 20 KeyValueStore::op_tp worker waiting -2281> 2014-05-19 18:37:54.950194 7fd0396a8700 20 OSD::command_tp worker waiting -2280> 2014-05-19 18:37:55.181063 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:00.270399) v2 ==== 47+0+0 (1007918699 0 0) 0x725da40 con 0x61b0160 -2279> 2014-05-19 18:37:55.181076 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:00.270399) v2 ==== 47+0+0 (1007918699 0 0) 0x60a16c0 con 0x31e1600 -2278> 2014-05-19 18:37:55.181117 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:00.270399) v2 -- ?+0 0x7122a80 con 0x61b0160 -2277> 2014-05-19 18:37:55.181152 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2276> 2014-05-19 18:37:55.181178 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:00.270399) v2 -- ?+0 0x51a6540 con 0x31e1600 -2275> 2014-05-19 18:37:55.181225 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2274> 2014-05-19 18:37:55.344624 7fd03c6ae700 20 OSD::op_tp worker waiting -2273> 2014-05-19 18:37:55.344650 7fd03bead700 20 OSD::op_tp worker waiting -2272> 2014-05-19 18:37:55.344672 7fd03f6b4700 20 OSD::op_tp worker waiting -2271> 2014-05-19 18:37:55.344696 7fd041eb9700 20 OSD::op_tp worker waiting -2270> 2014-05-19 18:37:55.344720 7fd03d6b0700 20 OSD::op_tp worker waiting -2269> 2014-05-19 18:37:55.344742 7fd0436bc700 20 OSD::op_tp worker waiting -2268> 2014-05-19 18:37:55.344767 7fd03e6b2700 20 OSD::op_tp worker waiting -2267> 2014-05-19 18:37:55.359210 7fd0446be700 20 OSD::op_tp worker waiting -2266> 2014-05-19 18:37:55.359233 7fd03feb5700 20 OSD::op_tp worker waiting -2265> 2014-05-19 18:37:55.366301 7fd040eb7700 20 OSD::op_tp worker waiting -2264> 2014-05-19 18:37:55.371752 7fd0406b6700 20 OSD::op_tp worker waiting -2263> 2014-05-19 18:37:55.393271 7fd039ea9700 20 OSD::disk_tp worker waiting -2262> 2014-05-19 18:37:55.396791 7fd03aeab700 20 OSD::op_tp worker waiting -2261> 2014-05-19 18:37:55.396817 7fd03eeb3700 20 OSD::op_tp worker waiting -2260> 2014-05-19 18:37:55.398436 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:55.398229) v2 ==== 47+0+0 (3160276996 0 0) 0x6efe540 con 0x4cb8420 -2259> 2014-05-19 18:37:55.398450 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:55.398229) v2 ==== 47+0+0 (3160276996 0 0) 0x6c4da40 con 0x4cb86e0 -2258> 2014-05-19 18:37:55.398482 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:55.398229) v2 -- ?+0 0x725da40 con 0x4cb8420 -2257> 2014-05-19 18:37:55.398519 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2256> 2014-05-19 18:37:55.398542 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:55.398229) v2 -- ?+0 0x60a16c0 con 0x4cb86e0 -2255> 2014-05-19 18:37:55.398587 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2254> 2014-05-19 18:37:55.414794 7fd042ebb700 20 OSD::op_tp worker waiting -2253> 2014-05-19 18:37:55.414817 7fd03deb1700 20 OSD::op_tp worker waiting -2252> 2014-05-19 18:37:55.423686 7fd04e104700 20 KeyValueStore::op_tp worker waiting -2251> 2014-05-19 18:37:55.423690 7fd03ceaf700 20 OSD::op_tp worker waiting -2250> 2014-05-19 18:37:55.423713 7fd0426ba700 20 OSD::op_tp worker waiting -2249> 2014-05-19 18:37:55.423734 7fd03b6ac700 20 OSD::op_tp worker waiting -2248> 2014-05-19 18:37:55.423760 7fd0416b8700 20 OSD::op_tp worker waiting -2247> 2014-05-19 18:37:55.501339 7fd04f907700 5 osd.0 21 tick -2246> 2014-05-19 18:37:55.501387 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.02 < max 0.5 = yes -2245> 2014-05-19 18:37:55.501401 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -2244> 2014-05-19 18:37:55.501408 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2243> 2014-05-19 18:37:55.501404 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 318.225 < min (86400 seconds) -2242> 2014-05-19 18:37:55.501422 7fd04f907700 20 osd.0 21 sched_scrub done -2241> 2014-05-19 18:37:55.501425 7fd04f907700 10 osd.0 21 do_waiters -- start -2240> 2014-05-19 18:37:55.501427 7fd04f907700 10 osd.0 21 do_waiters -- finish -2239> 2014-05-19 18:37:55.633045 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:58.690930) v2 ==== 47+0+0 (1229001731 0 0) 0x511c380 con 0x618fe40 -2238> 2014-05-19 18:37:55.633059 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:58.690930) v2 ==== 47+0+0 (1229001731 0 0) 0x7746fc0 con 0x61ae420 -2237> 2014-05-19 18:37:55.633104 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:58.690930) v2 -- ?+0 0x6efe540 con 0x618fe40 -2236> 2014-05-19 18:37:55.633137 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2235> 2014-05-19 18:37:55.633164 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:58.690930) v2 -- ?+0 0x6c4da40 con 0x61ae420 -2234> 2014-05-19 18:37:55.633192 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2233> 2014-05-19 18:37:55.784331 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:55.784135) v2 ==== 47+0+0 (1042547521 0 0) 0x36f7880 con 0x4cb82c0 -2232> 2014-05-19 18:37:55.784342 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:55.784135) v2 ==== 47+0+0 (1042547521 0 0) 0x656c540 con 0x4cb8160 -2231> 2014-05-19 18:37:55.784379 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:55.784135) v2 -- ?+0 0x511c380 con 0x4cb82c0 -2230> 2014-05-19 18:37:55.784417 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2229> 2014-05-19 18:37:55.784452 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:55.784135) v2 -- ?+0 0x7746fc0 con 0x4cb8160 -2228> 2014-05-19 18:37:55.784543 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2227> 2014-05-19 18:37:56.471762 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:59.529648) v2 ==== 47+0+0 (1660435674 0 0) 0x5deb500 con 0x4cb9340 -2226> 2014-05-19 18:37:56.471794 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:59.529648) v2 ==== 47+0+0 (1660435674 0 0) 0x60e0700 con 0x4cb8dc0 -2225> 2014-05-19 18:37:56.471813 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:59.529648) v2 -- ?+0 0x36f7880 con 0x4cb9340 -2224> 2014-05-19 18:37:56.471842 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2223> 2014-05-19 18:37:56.471866 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:59.529648) v2 -- ?+0 0x656c540 con 0x4cb8dc0 -2222> 2014-05-19 18:37:56.471958 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2221> 2014-05-19 18:37:56.501513 7fd04f907700 5 osd.0 21 tick -2220> 2014-05-19 18:37:56.501534 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2219> 2014-05-19 18:37:56.501539 7fd04f907700 10 osd.0 21 do_waiters -- start -2218> 2014-05-19 18:37:56.501541 7fd04f907700 10 osd.0 21 do_waiters -- finish -2217> 2014-05-19 18:37:56.501570 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2216> 2014-05-19 18:37:56.769400 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -2215> 2014-05-19 18:37:56.769418 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -2214> 2014-05-19 18:37:56.769426 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x860fa40 con 0x3622580 -2213> 2014-05-19 18:37:56.769451 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x7998fc0 con 0x3622840 -2212> 2014-05-19 18:37:56.769489 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x6e2a700 con 0x3622b00 -2211> 2014-05-19 18:37:56.769504 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x6009500 con 0x3622dc0 -2210> 2014-05-19 18:37:56.769517 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x77468c0 con 0x30594a0 -2209> 2014-05-19 18:37:56.769543 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x63781c0 con 0x3622f20 -2208> 2014-05-19 18:37:56.769576 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x7178c40 con 0x5e7ef20 -2207> 2014-05-19 18:37:56.769602 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x55f5c00 con 0x5e7f1e0 -2206> 2014-05-19 18:37:56.769627 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x7930540 con 0x5e7e2c0 -2205> 2014-05-19 18:37:56.769654 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x7122540 con 0x5d7bb80 -2204> 2014-05-19 18:37:56.769683 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x7aa4e00 con 0x5e7ec60 -2203> 2014-05-19 18:37:56.769708 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x63d9340 con 0x531b760 -2202> 2014-05-19 18:37:56.769733 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x55181c0 con 0x53bba20 -2201> 2014-05-19 18:37:56.769758 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x5c041c0 con 0x51cac60 -2200> 2014-05-19 18:37:56.769782 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x73c28c0 con 0x5bd66e0 -2199> 2014-05-19 18:37:56.769808 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:37:56.769424) v2 -- ?+0 0x55c3880 con 0x5bd6840 -2198> 2014-05-19 18:37:56.769808 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x6504700 con 0x3622580 -2197> 2014-05-19 18:37:56.769863 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2196> 2014-05-19 18:37:56.769886 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x72e5500 con 0x3622dc0 -2195> 2014-05-19 18:37:56.769955 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2194> 2014-05-19 18:37:56.769975 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x5f328c0 con 0x30594a0 -2193> 2014-05-19 18:37:56.770023 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2192> 2014-05-19 18:37:56.770039 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x711fa40 con 0x3622840 -2191> 2014-05-19 18:37:56.770069 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2190> 2014-05-19 18:37:56.770081 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x5ccc8c0 con 0x3622b00 -2189> 2014-05-19 18:37:56.770114 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2188> 2014-05-19 18:37:56.770129 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x7733dc0 con 0x5e7ef20 -2187> 2014-05-19 18:37:56.770154 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2186> 2014-05-19 18:37:56.770161 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x59bc540 con 0x5e7f1e0 -2185> 2014-05-19 18:37:56.770176 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2184> 2014-05-19 18:37:56.770181 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x7746c40 con 0x5e7e2c0 -2183> 2014-05-19 18:37:56.770209 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2182> 2014-05-19 18:37:56.770223 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x77e8c40 con 0x5d7bb80 -2181> 2014-05-19 18:37:56.770241 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2180> 2014-05-19 18:37:56.770250 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x6c4ca80 con 0x53bba20 -2179> 2014-05-19 18:37:56.770267 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2178> 2014-05-19 18:37:56.770290 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x5e28380 con 0x51cac60 -2177> 2014-05-19 18:37:56.770308 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2176> 2014-05-19 18:37:56.770314 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x3525880 con 0x5bd66e0 -2175> 2014-05-19 18:37:56.770328 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2174> 2014-05-19 18:37:56.770335 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 93 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x5cbc540 con 0x5bd6840 -2173> 2014-05-19 18:37:56.770349 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2172> 2014-05-19 18:37:56.770355 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x7980380 con 0x3622f20 -2171> 2014-05-19 18:37:56.770369 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2170> 2014-05-19 18:37:56.770377 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x798e1c0 con 0x5e7ec60 -2169> 2014-05-19 18:37:56.770405 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2168> 2014-05-19 18:37:56.770414 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:37:56.769424) v2 ==== 47+0+0 (3530801012 0 0) 0x66f9a40 con 0x531b760 -2167> 2014-05-19 18:37:56.770429 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2166> 2014-05-19 18:37:56.778126 7fd04e905700 20 KeyValueStore::op_tp worker waiting -2165> 2014-05-19 18:37:56.778127 7fd043ebd700 20 OSD::op_tp worker waiting -2164> 2014-05-19 18:37:56.950289 7fd0396a8700 20 OSD::command_tp worker waiting -2163> 2014-05-19 18:37:57.344702 7fd03c6ae700 20 OSD::op_tp worker waiting -2162> 2014-05-19 18:37:57.344722 7fd03bead700 20 OSD::op_tp worker waiting -2161> 2014-05-19 18:37:57.344766 7fd03f6b4700 20 OSD::op_tp worker waiting -2160> 2014-05-19 18:37:57.344789 7fd03d6b0700 20 OSD::op_tp worker waiting -2159> 2014-05-19 18:37:57.344810 7fd041eb9700 20 OSD::op_tp worker waiting -2158> 2014-05-19 18:37:57.344831 7fd0436bc700 20 OSD::op_tp worker waiting -2157> 2014-05-19 18:37:57.344857 7fd03e6b2700 20 OSD::op_tp worker waiting -2156> 2014-05-19 18:37:57.359301 7fd0446be700 20 OSD::op_tp worker waiting -2155> 2014-05-19 18:37:57.359324 7fd03feb5700 20 OSD::op_tp worker waiting -2154> 2014-05-19 18:37:57.366399 7fd040eb7700 20 OSD::op_tp worker waiting -2153> 2014-05-19 18:37:57.371832 7fd0406b6700 20 OSD::op_tp worker waiting -2152> 2014-05-19 18:37:57.393369 7fd039ea9700 20 OSD::disk_tp worker waiting -2151> 2014-05-19 18:37:57.396892 7fd03aeab700 20 OSD::op_tp worker waiting -2150> 2014-05-19 18:37:57.396920 7fd03eeb3700 20 OSD::op_tp worker waiting -2149> 2014-05-19 18:37:57.414874 7fd042ebb700 20 OSD::op_tp worker waiting -2148> 2014-05-19 18:37:57.414896 7fd03deb1700 20 OSD::op_tp worker waiting -2147> 2014-05-19 18:37:57.423787 7fd04e104700 20 KeyValueStore::op_tp worker waiting -2146> 2014-05-19 18:37:57.423791 7fd0426ba700 20 OSD::op_tp worker waiting -2145> 2014-05-19 18:37:57.423814 7fd03ceaf700 20 OSD::op_tp worker waiting -2144> 2014-05-19 18:37:57.423834 7fd0416b8700 20 OSD::op_tp worker waiting -2143> 2014-05-19 18:37:57.423855 7fd03b6ac700 20 OSD::op_tp worker waiting -2142> 2014-05-19 18:37:57.501627 7fd04f907700 5 osd.0 21 tick -2141> 2014-05-19 18:37:57.501644 7fd04f907700 7 osd.0 21 do_mon_report -2140> 2014-05-19 18:37:57.501647 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -2139> 2014-05-19 18:37:57.501651 7fd04f907700 20 osd.0 21 send_pg_stats -2138> 2014-05-19 18:37:57.501659 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2137> 2014-05-19 18:37:57.501663 7fd04f907700 10 osd.0 21 do_waiters -- start -2136> 2014-05-19 18:37:57.501665 7fd04f907700 10 osd.0 21 do_waiters -- finish -2135> 2014-05-19 18:37:57.501696 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2134> 2014-05-19 18:37:57.698941 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:57.698745) v2 ==== 47+0+0 (752000839 0 0) 0x6504a80 con 0x4cb8420 -2133> 2014-05-19 18:37:57.698945 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:57.698745) v2 ==== 47+0+0 (752000839 0 0) 0x5519340 con 0x4cb86e0 -2132> 2014-05-19 18:37:57.698991 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:57.698745) v2 -- ?+0 0x5deb500 con 0x4cb8420 -2131> 2014-05-19 18:37:57.699022 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2130> 2014-05-19 18:37:57.699070 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:57.698745) v2 -- ?+0 0x60e0700 con 0x4cb86e0 -2129> 2014-05-19 18:37:57.699131 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2128> 2014-05-19 18:37:58.395971 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:03.485385) v2 ==== 47+0+0 (607308078 0 0) 0x3524e00 con 0x5b21600 -2127> 2014-05-19 18:37:58.396020 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:03.485385) v2 -- ?+0 0x5519340 con 0x5b21600 -2126> 2014-05-19 18:37:58.395929 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:03.485385) v2 ==== 47+0+0 (607308078 0 0) 0x60e01c0 con 0x5f5b760 -2125> 2014-05-19 18:37:58.396098 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2124> 2014-05-19 18:37:58.396132 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:03.485385) v2 -- ?+0 0x6504a80 con 0x5f5b760 -2123> 2014-05-19 18:37:58.396174 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -2122> 2014-05-19 18:37:58.402261 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:01.460180) v2 ==== 47+0+0 (2895333089 0 0) 0x784d6c0 con 0x61ae160 -2121> 2014-05-19 18:37:58.402280 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:01.460180) v2 ==== 47+0+0 (2895333089 0 0) 0x71221c0 con 0x61b0420 -2120> 2014-05-19 18:37:58.402304 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:01.460180) v2 -- ?+0 0x3524e00 con 0x61ae160 -2119> 2014-05-19 18:37:58.402335 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2118> 2014-05-19 18:37:58.402359 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:01.460180) v2 -- ?+0 0x60e01c0 con 0x61b0420 -2117> 2014-05-19 18:37:58.402389 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2116> 2014-05-19 18:37:58.501747 7fd04f907700 5 osd.0 21 tick -2115> 2014-05-19 18:37:58.501796 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -2114> 2014-05-19 18:37:58.501809 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -2113> 2014-05-19 18:37:58.501816 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2112> 2014-05-19 18:37:58.501812 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 321.226 < min (86400 seconds) -2111> 2014-05-19 18:37:58.501830 7fd04f907700 20 osd.0 21 sched_scrub done -2110> 2014-05-19 18:37:58.501833 7fd04f907700 10 osd.0 21 do_waiters -- start -2109> 2014-05-19 18:37:58.501835 7fd04f907700 10 osd.0 21 do_waiters -- finish -2108> 2014-05-19 18:37:58.533540 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:01.591452) v2 ==== 47+0+0 (1909761090 0 0) 0x5bea1c0 con 0x61ae420 -2107> 2014-05-19 18:37:58.533561 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:01.591452) v2 ==== 47+0+0 (1909761090 0 0) 0x860f500 con 0x618fe40 -2106> 2014-05-19 18:37:58.533588 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:01.591452) v2 -- ?+0 0x784d6c0 con 0x61ae420 -2105> 2014-05-19 18:37:58.533620 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2104> 2014-05-19 18:37:58.533644 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:01.591452) v2 -- ?+0 0x71221c0 con 0x618fe40 -2103> 2014-05-19 18:37:58.533683 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -2102> 2014-05-19 18:37:58.684910 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:58.684705) v2 ==== 47+0+0 (1928194435 0 0) 0x656c1c0 con 0x4cb8160 -2101> 2014-05-19 18:37:58.684931 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:37:58.684705) v2 ==== 47+0+0 (1928194435 0 0) 0x36f7180 con 0x4cb82c0 -2100> 2014-05-19 18:37:58.684951 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:58.684705) v2 -- ?+0 0x5bea1c0 con 0x4cb8160 -2099> 2014-05-19 18:37:58.684981 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2098> 2014-05-19 18:37:58.685022 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:37:58.684705) v2 -- ?+0 0x860f500 con 0x4cb82c0 -2097> 2014-05-19 18:37:58.685063 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -2096> 2014-05-19 18:37:58.772249 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:01.830209) v2 ==== 47+0+0 (2317590890 0 0) 0x788a540 con 0x4cb8dc0 -2095> 2014-05-19 18:37:58.772267 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:01.830209) v2 ==== 47+0+0 (2317590890 0 0) 0x70ea540 con 0x4cb9340 -2094> 2014-05-19 18:37:58.772302 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:01.830209) v2 -- ?+0 0x656c1c0 con 0x4cb8dc0 -2093> 2014-05-19 18:37:58.772391 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2092> 2014-05-19 18:37:58.772444 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:01.830209) v2 -- ?+0 0x36f7180 con 0x4cb9340 -2091> 2014-05-19 18:37:58.772484 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -2090> 2014-05-19 18:37:58.778236 7fd04e905700 20 KeyValueStore::op_tp worker waiting -2089> 2014-05-19 18:37:58.778238 7fd043ebd700 20 OSD::op_tp worker waiting -2088> 2014-05-19 18:37:58.950384 7fd0396a8700 20 OSD::command_tp worker waiting -2087> 2014-05-19 18:37:59.281453 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:04.370943) v2 ==== 47+0+0 (1535294340 0 0) 0x725ca80 con 0x61b0160 -2086> 2014-05-19 18:37:59.281455 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:04.370943) v2 ==== 47+0+0 (1535294340 0 0) 0x60a1180 con 0x31e1600 -2085> 2014-05-19 18:37:59.281507 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.370943) v2 -- ?+0 0x70ea540 con 0x61b0160 -2084> 2014-05-19 18:37:59.281539 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2083> 2014-05-19 18:37:59.281607 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.370943) v2 -- ?+0 0x788a540 con 0x31e1600 -2082> 2014-05-19 18:37:59.281685 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -2081> 2014-05-19 18:37:59.320700 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:04.410199) v2 ==== 47+0+0 (501864937 0 0) 0x2e48e00 con 0x3094160 -2080> 2014-05-19 18:37:59.320702 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 89 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:04.410199) v2 ==== 47+0+0 (501864937 0 0) 0x3438380 con 0x30946e0 -2079> 2014-05-19 18:37:59.320756 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.410199) v2 -- ?+0 0x725ca80 con 0x3094160 -2078> 2014-05-19 18:37:59.320789 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2077> 2014-05-19 18:37:59.320834 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.410199) v2 -- ?+0 0x60a1180 con 0x30946e0 -2076> 2014-05-19 18:37:59.320877 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -2075> 2014-05-19 18:37:59.344801 7fd03bead700 20 OSD::op_tp worker waiting -2074> 2014-05-19 18:37:59.344824 7fd03c6ae700 20 OSD::op_tp worker waiting -2073> 2014-05-19 18:37:59.344843 7fd03f6b4700 20 OSD::op_tp worker waiting -2072> 2014-05-19 18:37:59.344862 7fd03d6b0700 20 OSD::op_tp worker waiting -2071> 2014-05-19 18:37:59.344881 7fd041eb9700 20 OSD::op_tp worker waiting -2070> 2014-05-19 18:37:59.344900 7fd0436bc700 20 OSD::op_tp worker waiting -2069> 2014-05-19 18:37:59.344962 7fd03e6b2700 20 OSD::op_tp worker waiting -2068> 2014-05-19 18:37:59.359389 7fd0446be700 20 OSD::op_tp worker waiting -2067> 2014-05-19 18:37:59.359410 7fd03feb5700 20 OSD::op_tp worker waiting -2066> 2014-05-19 18:37:59.366497 7fd040eb7700 20 OSD::op_tp worker waiting -2065> 2014-05-19 18:37:59.371912 7fd0406b6700 20 OSD::op_tp worker waiting -2064> 2014-05-19 18:37:59.393464 7fd039ea9700 20 OSD::disk_tp worker waiting -2063> 2014-05-19 18:37:59.396991 7fd03aeab700 20 OSD::op_tp worker waiting -2062> 2014-05-19 18:37:59.397014 7fd03eeb3700 20 OSD::op_tp worker waiting -2061> 2014-05-19 18:37:59.414972 7fd042ebb700 20 OSD::op_tp worker waiting -2060> 2014-05-19 18:37:59.414997 7fd03deb1700 20 OSD::op_tp worker waiting -2059> 2014-05-19 18:37:59.423886 7fd04e104700 20 KeyValueStore::op_tp worker waiting -2058> 2014-05-19 18:37:59.423889 7fd0426ba700 20 OSD::op_tp worker waiting -2057> 2014-05-19 18:37:59.423925 7fd03b6ac700 20 OSD::op_tp worker waiting -2056> 2014-05-19 18:37:59.423944 7fd0416b8700 20 OSD::op_tp worker waiting -2055> 2014-05-19 18:37:59.423967 7fd03ceaf700 20 OSD::op_tp worker waiting -2054> 2014-05-19 18:37:59.501924 7fd04f907700 5 osd.0 21 tick -2053> 2014-05-19 18:37:59.501972 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -2052> 2014-05-19 18:37:59.501985 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -2051> 2014-05-19 18:37:59.501993 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2050> 2014-05-19 18:37:59.501989 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 322.226 < min (86400 seconds) -2049> 2014-05-19 18:37:59.502007 7fd04f907700 20 osd.0 21 sched_scrub done -2048> 2014-05-19 18:37:59.502010 7fd04f907700 10 osd.0 21 do_waiters -- start -2047> 2014-05-19 18:37:59.502012 7fd04f907700 10 osd.0 21 do_waiters -- finish -2046> 2014-05-19 18:37:59.908225 7fd0456c0700 10 monclient: tick -2045> 2014-05-19 18:37:59.908238 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:37:29.908237) -2044> 2014-05-19 18:37:59.908255 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:37:59.908254; renew after: 2014-05-19 18:38:19.906664) -- no -2043> 2014-05-19 18:38:00.502078 7fd04f907700 5 osd.0 21 tick -2042> 2014-05-19 18:38:00.502097 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2041> 2014-05-19 18:38:00.502101 7fd04f907700 10 osd.0 21 do_waiters -- start -2040> 2014-05-19 18:38:00.502103 7fd04f907700 10 osd.0 21 do_waiters -- finish -2039> 2014-05-19 18:38:00.502134 7fd03a6aa700 20 OSD::recovery_tp worker waiting -2038> 2014-05-19 18:38:00.599449 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:00.599288) v2 ==== 47+0+0 (847463014 0 0) 0x55188c0 con 0x4cb86e0 -2037> 2014-05-19 18:38:00.599494 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:00.599288) v2 -- ?+0 0x3438380 con 0x4cb86e0 -2036> 2014-05-19 18:38:00.599532 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2035> 2014-05-19 18:38:00.599504 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:00.599288) v2 ==== 47+0+0 (847463014 0 0) 0x60a0c40 con 0x4cb8420 -2034> 2014-05-19 18:38:00.599548 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:00.599288) v2 -- ?+0 0x2e48e00 con 0x4cb8420 -2033> 2014-05-19 18:38:00.599634 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -2032> 2014-05-19 18:38:00.702658 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:03.760657) v2 ==== 47+0+0 (597006784 0 0) 0x784c380 con 0x61ae160 -2031> 2014-05-19 18:38:00.702699 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:03.760657) v2 -- ?+0 0x55188c0 con 0x61ae160 -2030> 2014-05-19 18:38:00.702739 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:03.760657) v2 ==== 47+0+0 (597006784 0 0) 0x64afc00 con 0x61b0420 -2029> 2014-05-19 18:38:00.702796 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2028> 2014-05-19 18:38:00.702856 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:03.760657) v2 -- ?+0 0x60a0c40 con 0x61b0420 -2027> 2014-05-19 18:38:00.702896 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -2026> 2014-05-19 18:38:00.778326 7fd04e905700 20 KeyValueStore::op_tp worker waiting -2025> 2014-05-19 18:38:00.778328 7fd043ebd700 20 OSD::op_tp worker waiting -2024> 2014-05-19 18:38:00.950480 7fd0396a8700 20 OSD::command_tp worker waiting -2023> 2014-05-19 18:38:01.344879 7fd03bead700 20 OSD::op_tp worker waiting -2022> 2014-05-19 18:38:01.344904 7fd03f6b4700 20 OSD::op_tp worker waiting -2021> 2014-05-19 18:38:01.344942 7fd03d6b0700 20 OSD::op_tp worker waiting -2020> 2014-05-19 18:38:01.344964 7fd03c6ae700 20 OSD::op_tp worker waiting -2019> 2014-05-19 18:38:01.344985 7fd0436bc700 20 OSD::op_tp worker waiting -2018> 2014-05-19 18:38:01.345031 7fd041eb9700 20 OSD::op_tp worker waiting -2017> 2014-05-19 18:38:01.345058 7fd03e6b2700 20 OSD::op_tp worker waiting -2016> 2014-05-19 18:38:01.359475 7fd0446be700 20 OSD::op_tp worker waiting -2015> 2014-05-19 18:38:01.359500 7fd03feb5700 20 OSD::op_tp worker waiting -2014> 2014-05-19 18:38:01.366595 7fd040eb7700 20 OSD::op_tp worker waiting -2013> 2014-05-19 18:38:01.371996 7fd0406b6700 20 OSD::op_tp worker waiting -2012> 2014-05-19 18:38:01.393558 7fd039ea9700 20 OSD::disk_tp worker waiting -2011> 2014-05-19 18:38:01.397089 7fd03aeab700 20 OSD::op_tp worker waiting -2010> 2014-05-19 18:38:01.397116 7fd03eeb3700 20 OSD::op_tp worker waiting -2009> 2014-05-19 18:38:01.415053 7fd042ebb700 20 OSD::op_tp worker waiting -2008> 2014-05-19 18:38:01.415073 7fd03deb1700 20 OSD::op_tp worker waiting -2007> 2014-05-19 18:38:01.423995 7fd04e104700 20 KeyValueStore::op_tp worker waiting -2006> 2014-05-19 18:38:01.423999 7fd0426ba700 20 OSD::op_tp worker waiting -2005> 2014-05-19 18:38:01.424036 7fd03b6ac700 20 OSD::op_tp worker waiting -2004> 2014-05-19 18:38:01.424054 7fd0416b8700 20 OSD::op_tp worker waiting -2003> 2014-05-19 18:38:01.424108 7fd03ceaf700 20 OSD::op_tp worker waiting -2002> 2014-05-19 18:38:01.502186 7fd04f907700 5 osd.0 21 tick -2001> 2014-05-19 18:38:01.502207 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -2000> 2014-05-19 18:38:01.502212 7fd04f907700 10 osd.0 21 do_waiters -- start -1999> 2014-05-19 18:38:01.502214 7fd04f907700 10 osd.0 21 do_waiters -- finish -1998> 2014-05-19 18:38:01.502255 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1997> 2014-05-19 18:38:01.581891 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:06.671469) v2 ==== 47+0+0 (3295058626 0 0) 0x60a0e00 con 0x31e1600 -1996> 2014-05-19 18:38:01.581897 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:06.671469) v2 ==== 47+0+0 (3295058626 0 0) 0x5c04e00 con 0x61b0160 -1995> 2014-05-19 18:38:01.581950 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:06.671469) v2 -- ?+0 0x784c380 con 0x31e1600 -1994> 2014-05-19 18:38:01.581980 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1993> 2014-05-19 18:38:01.582001 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:06.671469) v2 -- ?+0 0x64afc00 con 0x61b0160 -1992> 2014-05-19 18:38:01.582045 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1991> 2014-05-19 18:38:01.672829 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:04.730819) v2 ==== 47+0+0 (1703299275 0 0) 0x788b180 con 0x4cb8dc0 -1990> 2014-05-19 18:38:01.672863 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:04.730819) v2 ==== 47+0+0 (1703299275 0 0) 0x6e188c0 con 0x4cb9340 -1989> 2014-05-19 18:38:01.672960 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.730819) v2 -- ?+0 0x60a0e00 con 0x4cb8dc0 -1988> 2014-05-19 18:38:01.672990 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1987> 2014-05-19 18:38:01.673014 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.730819) v2 -- ?+0 0x5c04e00 con 0x4cb9340 -1986> 2014-05-19 18:38:01.673067 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1985> 2014-05-19 18:38:02.502296 7fd04f907700 5 osd.0 21 tick -1984> 2014-05-19 18:38:02.502312 7fd04f907700 7 osd.0 21 do_mon_report -1983> 2014-05-19 18:38:02.502316 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -1982> 2014-05-19 18:38:02.502320 7fd04f907700 20 osd.0 21 send_pg_stats -1981> 2014-05-19 18:38:02.502328 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1980> 2014-05-19 18:38:02.502331 7fd04f907700 10 osd.0 21 do_waiters -- start -1979> 2014-05-19 18:38:02.502333 7fd04f907700 10 osd.0 21 do_waiters -- finish -1978> 2014-05-19 18:38:02.502372 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1977> 2014-05-19 18:38:02.669936 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -1976> 2014-05-19 18:38:02.669954 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -1975> 2014-05-19 18:38:02.669962 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x7999180 con 0x3622580 -1974> 2014-05-19 18:38:02.669988 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x6e2a380 con 0x3622840 -1973> 2014-05-19 18:38:02.670027 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x511ddc0 con 0x3622b00 -1972> 2014-05-19 18:38:02.670041 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x3510380 con 0x3622dc0 -1971> 2014-05-19 18:38:02.670069 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x59bcc40 con 0x30594a0 -1970> 2014-05-19 18:38:02.670096 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x77321c0 con 0x3622f20 -1969> 2014-05-19 18:38:02.670125 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x5ccc1c0 con 0x5e7ef20 -1968> 2014-05-19 18:38:02.670151 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x711e8c0 con 0x5e7f1e0 -1967> 2014-05-19 18:38:02.670178 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x72e4c40 con 0x5e7e2c0 -1966> 2014-05-19 18:38:02.670202 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x5f7a8c0 con 0x5d7bb80 -1965> 2014-05-19 18:38:02.670228 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x662b500 con 0x5e7ec60 -1964> 2014-05-19 18:38:02.670253 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x7aa5180 con 0x531b760 -1963> 2014-05-19 18:38:02.670280 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x5497a40 con 0x53bba20 -1962> 2014-05-19 18:38:02.670303 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x6008380 con 0x51cac60 -1961> 2014-05-19 18:38:02.670328 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x63d9dc0 con 0x5bd66e0 -1960> 2014-05-19 18:38:02.670333 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x67101c0 con 0x3622580 -1959> 2014-05-19 18:38:02.670355 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:02.669961) v2 -- ?+0 0x6c4d500 con 0x5bd6840 -1958> 2014-05-19 18:38:02.670397 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1957> 2014-05-19 18:38:02.670432 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x711ee00 con 0x3622840 -1956> 2014-05-19 18:38:02.670465 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1955> 2014-05-19 18:38:02.670483 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x691b6c0 con 0x3622b00 -1954> 2014-05-19 18:38:02.670502 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1953> 2014-05-19 18:38:02.670508 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x8388700 con 0x3622dc0 -1952> 2014-05-19 18:38:02.670527 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1951> 2014-05-19 18:38:02.670537 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x7999dc0 con 0x30594a0 -1950> 2014-05-19 18:38:02.670553 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1949> 2014-05-19 18:38:02.670664 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x77e81c0 con 0x5d7bb80 -1948> 2014-05-19 18:38:02.670704 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1947> 2014-05-19 18:38:02.670741 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x59bcfc0 con 0x5e7f1e0 -1946> 2014-05-19 18:38:02.670765 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1945> 2014-05-19 18:38:02.670772 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x7747880 con 0x5e7e2c0 -1944> 2014-05-19 18:38:02.670786 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1943> 2014-05-19 18:38:02.670792 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x798f880 con 0x5e7ec60 -1942> 2014-05-19 18:38:02.670806 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1941> 2014-05-19 18:38:02.670811 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x66f8000 con 0x531b760 -1940> 2014-05-19 18:38:02.670825 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1939> 2014-05-19 18:38:02.670832 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x5e28c40 con 0x51cac60 -1938> 2014-05-19 18:38:02.670866 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1937> 2014-05-19 18:38:02.670872 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x70a8e00 con 0x5bd66e0 -1936> 2014-05-19 18:38:02.670898 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1935> 2014-05-19 18:38:02.670920 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x79801c0 con 0x3622f20 -1934> 2014-05-19 18:38:02.670943 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1933> 2014-05-19 18:38:02.670956 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x7733a40 con 0x5e7ef20 -1932> 2014-05-19 18:38:02.670975 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1931> 2014-05-19 18:38:02.670980 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x6c4c000 con 0x53bba20 -1930> 2014-05-19 18:38:02.670997 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1929> 2014-05-19 18:38:02.671009 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 94 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:02.669961) v2 ==== 47+0+0 (3817149481 0 0) 0x6113500 con 0x5bd6840 -1928> 2014-05-19 18:38:02.671026 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1927> 2014-05-19 18:38:02.778426 7fd04e905700 20 KeyValueStore::op_tp worker waiting -1926> 2014-05-19 18:38:02.778432 7fd043ebd700 20 OSD::op_tp worker waiting -1925> 2014-05-19 18:38:02.950572 7fd0396a8700 20 OSD::command_tp worker waiting -1924> 2014-05-19 18:38:03.344960 7fd03bead700 20 OSD::op_tp worker waiting -1923> 2014-05-19 18:38:03.344980 7fd03f6b4700 20 OSD::op_tp worker waiting -1922> 2014-05-19 18:38:03.345017 7fd03d6b0700 20 OSD::op_tp worker waiting -1921> 2014-05-19 18:38:03.345039 7fd03c6ae700 20 OSD::op_tp worker waiting -1920> 2014-05-19 18:38:03.345050 7fd0436bc700 20 OSD::op_tp worker waiting -1919> 2014-05-19 18:38:03.345111 7fd041eb9700 20 OSD::op_tp worker waiting -1918> 2014-05-19 18:38:03.345134 7fd03e6b2700 20 OSD::op_tp worker waiting -1917> 2014-05-19 18:38:03.359557 7fd0446be700 20 OSD::op_tp worker waiting -1916> 2014-05-19 18:38:03.359578 7fd03feb5700 20 OSD::op_tp worker waiting -1915> 2014-05-19 18:38:03.366693 7fd040eb7700 20 OSD::op_tp worker waiting -1914> 2014-05-19 18:38:03.372077 7fd0406b6700 20 OSD::op_tp worker waiting -1913> 2014-05-19 18:38:03.393655 7fd039ea9700 20 OSD::disk_tp worker waiting -1912> 2014-05-19 18:38:03.397187 7fd03aeab700 20 OSD::op_tp worker waiting -1911> 2014-05-19 18:38:03.397210 7fd03eeb3700 20 OSD::op_tp worker waiting -1910> 2014-05-19 18:38:03.415127 7fd042ebb700 20 OSD::op_tp worker waiting -1909> 2014-05-19 18:38:03.415151 7fd03deb1700 20 OSD::op_tp worker waiting -1908> 2014-05-19 18:38:03.424104 7fd04e104700 20 KeyValueStore::op_tp worker waiting -1907> 2014-05-19 18:38:03.424107 7fd0426ba700 20 OSD::op_tp worker waiting -1906> 2014-05-19 18:38:03.424143 7fd03b6ac700 20 OSD::op_tp worker waiting -1905> 2014-05-19 18:38:03.424162 7fd0416b8700 20 OSD::op_tp worker waiting -1904> 2014-05-19 18:38:03.424191 7fd03ceaf700 20 OSD::op_tp worker waiting -1903> 2014-05-19 18:38:03.502418 7fd04f907700 5 osd.0 21 tick -1902> 2014-05-19 18:38:03.502440 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1901> 2014-05-19 18:38:03.502444 7fd04f907700 10 osd.0 21 do_waiters -- start -1900> 2014-05-19 18:38:03.502447 7fd04f907700 10 osd.0 21 do_waiters -- finish -1899> 2014-05-19 18:38:03.502487 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1898> 2014-05-19 18:38:03.603265 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:06.661227) v2 ==== 47+0+0 (134363694 0 0) 0x64aec40 con 0x61b0420 -1897> 2014-05-19 18:38:03.603285 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:06.661227) v2 ==== 47+0+0 (134363694 0 0) 0x669f340 con 0x61ae160 -1896> 2014-05-19 18:38:03.603315 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:06.661227) v2 -- ?+0 0x6e188c0 con 0x61b0420 -1895> 2014-05-19 18:38:03.603350 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1894> 2014-05-19 18:38:03.603393 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:06.661227) v2 -- ?+0 0x788b180 con 0x61ae160 -1893> 2014-05-19 18:38:03.603436 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1892> 2014-05-19 18:38:03.696214 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:08.785873) v2 ==== 47+0+0 (794882040 0 0) 0x60e0700 con 0x5f5b760 -1891> 2014-05-19 18:38:03.696228 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:08.785873) v2 ==== 47+0+0 (794882040 0 0) 0x3524c40 con 0x5b21600 -1890> 2014-05-19 18:38:03.696268 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:08.785873) v2 -- ?+0 0x669f340 con 0x5b21600 -1889> 2014-05-19 18:38:03.696297 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1888> 2014-05-19 18:38:03.696322 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:08.785873) v2 -- ?+0 0x64aec40 con 0x5f5b760 -1887> 2014-05-19 18:38:03.696360 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1886> 2014-05-19 18:38:04.434013 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:07.492019) v2 ==== 47+0+0 (1372918466 0 0) 0x5bea380 con 0x61ae420 -1885> 2014-05-19 18:38:04.434062 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:07.492019) v2 -- ?+0 0x3524c40 con 0x61ae420 -1884> 2014-05-19 18:38:04.434094 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1883> 2014-05-19 18:38:04.434113 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:07.492019) v2 ==== 47+0+0 (1372918466 0 0) 0x860e000 con 0x618fe40 -1882> 2014-05-19 18:38:04.434145 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:07.492019) v2 -- ?+0 0x60e0700 con 0x618fe40 -1881> 2014-05-19 18:38:04.434179 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1880> 2014-05-19 18:38:04.482268 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:09.571980) v2 ==== 47+0+0 (3345824945 0 0) 0x5c05880 con 0x61b0160 -1879> 2014-05-19 18:38:04.482283 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:09.571980) v2 ==== 47+0+0 (3345824945 0 0) 0x60a0540 con 0x31e1600 -1878> 2014-05-19 18:38:04.482318 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.571980) v2 -- ?+0 0x860e000 con 0x61b0160 -1877> 2014-05-19 18:38:04.482351 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1876> 2014-05-19 18:38:04.482382 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.571980) v2 -- ?+0 0x5bea380 con 0x31e1600 -1875> 2014-05-19 18:38:04.482420 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1874> 2014-05-19 18:38:04.502534 7fd04f907700 5 osd.0 21 tick -1873> 2014-05-19 18:38:04.502585 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1872> 2014-05-19 18:38:04.502582 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -1871> 2014-05-19 18:38:04.502595 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -1870> 2014-05-19 18:38:04.502598 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 327.227 < min (86400 seconds) -1869> 2014-05-19 18:38:04.502616 7fd04f907700 20 osd.0 21 sched_scrub done -1868> 2014-05-19 18:38:04.502619 7fd04f907700 10 osd.0 21 do_waiters -- start -1867> 2014-05-19 18:38:04.502621 7fd04f907700 10 osd.0 21 do_waiters -- finish -1866> 2014-05-19 18:38:04.585502 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:04.585301) v2 ==== 47+0+0 (4050639926 0 0) 0x656c8c0 con 0x4cb8160 -1865> 2014-05-19 18:38:04.585519 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:04.585301) v2 ==== 47+0+0 (4050639926 0 0) 0x36f6000 con 0x4cb82c0 -1864> 2014-05-19 18:38:04.585541 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.585301) v2 -- ?+0 0x60a0540 con 0x4cb8160 -1863> 2014-05-19 18:38:04.585570 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1862> 2014-05-19 18:38:04.585589 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.585301) v2 -- ?+0 0x5c05880 con 0x4cb82c0 -1861> 2014-05-19 18:38:04.585622 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1860> 2014-05-19 18:38:04.621008 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:09.710735) v2 ==== 47+0+0 (421087978 0 0) 0x6543a40 con 0x30946e0 -1859> 2014-05-19 18:38:04.621015 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 90 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:09.710735) v2 ==== 47+0+0 (421087978 0 0) 0x860e540 con 0x3094160 -1858> 2014-05-19 18:38:04.621058 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.710735) v2 -- ?+0 0x656c8c0 con 0x30946e0 -1857> 2014-05-19 18:38:04.621088 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1856> 2014-05-19 18:38:04.621109 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.710735) v2 -- ?+0 0x36f6000 con 0x3094160 -1855> 2014-05-19 18:38:04.621152 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1854> 2014-05-19 18:38:04.778524 7fd04e905700 20 KeyValueStore::op_tp worker waiting -1853> 2014-05-19 18:38:04.778527 7fd043ebd700 20 OSD::op_tp worker waiting -1852> 2014-05-19 18:38:04.950663 7fd0396a8700 20 OSD::command_tp worker waiting -1851> 2014-05-19 18:38:04.970480 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -1850> 2014-05-19 18:38:04.970498 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -1849> 2014-05-19 18:38:04.970506 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x66f9500 con 0x3622580 -1848> 2014-05-19 18:38:04.970531 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x77e8e00 con 0x3622840 -1847> 2014-05-19 18:38:04.970569 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x77336c0 con 0x3622b00 -1846> 2014-05-19 18:38:04.970583 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x6d616c0 con 0x3622dc0 -1845> 2014-05-19 18:38:04.970597 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x6b70540 con 0x30594a0 -1844> 2014-05-19 18:38:04.970621 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x59bd500 con 0x3622f20 -1843> 2014-05-19 18:38:04.970649 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x72e5dc0 con 0x5e7ef20 -1842> 2014-05-19 18:38:04.970673 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x692cc40 con 0x5e7f1e0 -1841> 2014-05-19 18:38:04.970696 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x5f7ac40 con 0x5e7e2c0 -1840> 2014-05-19 18:38:04.970723 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x860f6c0 con 0x5d7bb80 -1839> 2014-05-19 18:38:04.970750 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x7aa4700 con 0x5e7ec60 -1838> 2014-05-19 18:38:04.970780 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x63d9180 con 0x531b760 -1837> 2014-05-19 18:38:04.970812 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x79e6c40 con 0x53bba20 -1836> 2014-05-19 18:38:04.970838 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x5c05500 con 0x51cac60 -1835> 2014-05-19 18:38:04.970858 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x66f8380 con 0x5bd66e0 -1834> 2014-05-19 18:38:04.970872 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:04.970504) v2 -- ?+0 0x662b6c0 con 0x5bd6840 -1833> 2014-05-19 18:38:04.970893 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x6711180 con 0x3622580 -1832> 2014-05-19 18:38:04.970951 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1831> 2014-05-19 18:38:04.970975 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x72e48c0 con 0x3622b00 -1830> 2014-05-19 18:38:04.971003 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1829> 2014-05-19 18:38:04.971014 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x7a5e1c0 con 0x3622840 -1828> 2014-05-19 18:38:04.971035 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1827> 2014-05-19 18:38:04.971047 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x8388540 con 0x3622dc0 -1826> 2014-05-19 18:38:04.971068 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1825> 2014-05-19 18:38:04.971075 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x51a6a80 con 0x5e7ef20 -1824> 2014-05-19 18:38:04.971105 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1823> 2014-05-19 18:38:04.971139 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x682f340 con 0x3622f20 -1822> 2014-05-19 18:38:04.971173 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1821> 2014-05-19 18:38:04.971185 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x59bddc0 con 0x5e7f1e0 -1820> 2014-05-19 18:38:04.971203 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1819> 2014-05-19 18:38:04.971209 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x7746540 con 0x5e7e2c0 -1818> 2014-05-19 18:38:04.971223 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1817> 2014-05-19 18:38:04.971228 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x8649a40 con 0x5d7bb80 -1816> 2014-05-19 18:38:04.971241 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1815> 2014-05-19 18:38:04.971292 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x66f8700 con 0x531b760 -1814> 2014-05-19 18:38:04.971326 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1813> 2014-05-19 18:38:04.971336 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x6c4cc40 con 0x53bba20 -1812> 2014-05-19 18:38:04.971353 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1811> 2014-05-19 18:38:04.971361 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x5e29340 con 0x51cac60 -1810> 2014-05-19 18:38:04.971376 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1809> 2014-05-19 18:38:04.971387 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x6113880 con 0x5bd6840 -1808> 2014-05-19 18:38:04.971401 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1807> 2014-05-19 18:38:04.971409 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x7998a80 con 0x30594a0 -1806> 2014-05-19 18:38:04.971424 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1805> 2014-05-19 18:38:04.971430 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x798fdc0 con 0x5e7ec60 -1804> 2014-05-19 18:38:04.971446 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1803> 2014-05-19 18:38:04.971453 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 95 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:04.970504) v2 ==== 47+0+0 (3213681633 0 0) 0x70a8540 con 0x5bd66e0 -1802> 2014-05-19 18:38:04.971469 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1801> 2014-05-19 18:38:05.345049 7fd03f6b4700 20 OSD::op_tp worker waiting -1800> 2014-05-19 18:38:05.345092 7fd03d6b0700 20 OSD::op_tp worker waiting -1799> 2014-05-19 18:38:05.345114 7fd0436bc700 20 OSD::op_tp worker waiting -1798> 2014-05-19 18:38:05.345134 7fd03bead700 20 OSD::op_tp worker waiting -1797> 2014-05-19 18:38:05.345156 7fd03c6ae700 20 OSD::op_tp worker waiting -1796> 2014-05-19 18:38:05.345180 7fd041eb9700 20 OSD::op_tp worker waiting -1795> 2014-05-19 18:38:05.345207 7fd03e6b2700 20 OSD::op_tp worker waiting -1794> 2014-05-19 18:38:05.359638 7fd0446be700 20 OSD::op_tp worker waiting -1793> 2014-05-19 18:38:05.359656 7fd03feb5700 20 OSD::op_tp worker waiting -1792> 2014-05-19 18:38:05.366792 7fd040eb7700 20 OSD::op_tp worker waiting -1791> 2014-05-19 18:38:05.372161 7fd0406b6700 20 OSD::op_tp worker waiting -1790> 2014-05-19 18:38:05.393750 7fd039ea9700 20 OSD::disk_tp worker waiting -1789> 2014-05-19 18:38:05.397285 7fd03aeab700 20 OSD::op_tp worker waiting -1788> 2014-05-19 18:38:05.397313 7fd03eeb3700 20 OSD::op_tp worker waiting -1787> 2014-05-19 18:38:05.415207 7fd042ebb700 20 OSD::op_tp worker waiting -1786> 2014-05-19 18:38:05.415229 7fd03deb1700 20 OSD::op_tp worker waiting -1785> 2014-05-19 18:38:05.424218 7fd04e104700 20 KeyValueStore::op_tp worker waiting -1784> 2014-05-19 18:38:05.424221 7fd0426ba700 20 OSD::op_tp worker waiting -1783> 2014-05-19 18:38:05.424259 7fd03b6ac700 20 OSD::op_tp worker waiting -1782> 2014-05-19 18:38:05.424280 7fd0416b8700 20 OSD::op_tp worker waiting -1781> 2014-05-19 18:38:05.424331 7fd03ceaf700 20 OSD::op_tp worker waiting -1780> 2014-05-19 18:38:05.502706 7fd04f907700 5 osd.0 21 tick -1779> 2014-05-19 18:38:05.502727 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1778> 2014-05-19 18:38:05.502732 7fd04f907700 10 osd.0 21 do_waiters -- start -1777> 2014-05-19 18:38:05.502734 7fd04f907700 10 osd.0 21 do_waiters -- finish -1776> 2014-05-19 18:38:05.502774 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1775> 2014-05-19 18:38:06.500012 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:06.499823) v2 ==== 47+0+0 (2417460013 0 0) 0x36f7500 con 0x4cb86e0 -1774> 2014-05-19 18:38:06.500018 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:06.499823) v2 ==== 47+0+0 (2417460013 0 0) 0x3524540 con 0x4cb8420 -1773> 2014-05-19 18:38:06.500075 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:06.499823) v2 -- ?+0 0x6543a40 con 0x4cb86e0 -1772> 2014-05-19 18:38:06.500164 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1771> 2014-05-19 18:38:06.500192 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:06.499823) v2 -- ?+0 0x860e540 con 0x4cb8420 -1770> 2014-05-19 18:38:06.500225 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1769> 2014-05-19 18:38:06.502795 7fd04f907700 5 osd.0 21 tick -1768> 2014-05-19 18:38:06.502813 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1767> 2014-05-19 18:38:06.502818 7fd04f907700 10 osd.0 21 do_waiters -- start -1766> 2014-05-19 18:38:06.502821 7fd04f907700 10 osd.0 21 do_waiters -- finish -1765> 2014-05-19 18:38:06.502849 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1764> 2014-05-19 18:38:06.734593 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:09.792570) v2 ==== 47+0+0 (1490741229 0 0) 0x653e8c0 con 0x618fe40 -1763> 2014-05-19 18:38:06.734640 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.792570) v2 -- ?+0 0x3524540 con 0x618fe40 -1762> 2014-05-19 18:38:06.734644 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:09.792570) v2 ==== 47+0+0 (1490741229 0 0) 0x70eb340 con 0x61ae420 -1761> 2014-05-19 18:38:06.734673 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1760> 2014-05-19 18:38:06.734695 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.792570) v2 -- ?+0 0x36f7500 con 0x61ae420 -1759> 2014-05-19 18:38:06.734741 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1758> 2014-05-19 18:38:06.778628 7fd04e905700 20 KeyValueStore::op_tp worker waiting -1757> 2014-05-19 18:38:06.778630 7fd043ebd700 20 OSD::op_tp worker waiting -1756> 2014-05-19 18:38:06.886123 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:06.885955) v2 ==== 47+0+0 (2292344505 0 0) 0x36f68c0 con 0x4cb82c0 -1755> 2014-05-19 18:38:06.886146 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:06.885955) v2 ==== 47+0+0 (2292344505 0 0) 0x656cfc0 con 0x4cb8160 -1754> 2014-05-19 18:38:06.886162 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:06.885955) v2 -- ?+0 0x653e8c0 con 0x4cb82c0 -1753> 2014-05-19 18:38:06.886271 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1752> 2014-05-19 18:38:06.886302 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:06.885955) v2 -- ?+0 0x70eb340 con 0x4cb8160 -1751> 2014-05-19 18:38:06.886330 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1750> 2014-05-19 18:38:06.950763 7fd0396a8700 20 OSD::command_tp worker waiting -1749> 2014-05-19 18:38:07.345147 7fd03f6b4700 20 OSD::op_tp worker waiting -1748> 2014-05-19 18:38:07.345169 7fd03d6b0700 20 OSD::op_tp worker waiting -1747> 2014-05-19 18:38:07.345181 7fd0436bc700 20 OSD::op_tp worker waiting -1746> 2014-05-19 18:38:07.345204 7fd03bead700 20 OSD::op_tp worker waiting -1745> 2014-05-19 18:38:07.345229 7fd03c6ae700 20 OSD::op_tp worker waiting -1744> 2014-05-19 18:38:07.345253 7fd041eb9700 20 OSD::op_tp worker waiting -1743> 2014-05-19 18:38:07.345269 7fd03e6b2700 20 OSD::op_tp worker waiting -1742> 2014-05-19 18:38:07.359719 7fd0446be700 20 OSD::op_tp worker waiting -1741> 2014-05-19 18:38:07.359743 7fd03feb5700 20 OSD::op_tp worker waiting -1740> 2014-05-19 18:38:07.366889 7fd040eb7700 20 OSD::op_tp worker waiting -1739> 2014-05-19 18:38:07.372255 7fd0406b6700 20 OSD::op_tp worker waiting -1738> 2014-05-19 18:38:07.393848 7fd039ea9700 20 OSD::disk_tp worker waiting -1737> 2014-05-19 18:38:07.397386 7fd03aeab700 20 OSD::op_tp worker waiting -1736> 2014-05-19 18:38:07.397411 7fd03eeb3700 20 OSD::op_tp worker waiting -1735> 2014-05-19 18:38:07.415288 7fd042ebb700 20 OSD::op_tp worker waiting -1734> 2014-05-19 18:38:07.415310 7fd03deb1700 20 OSD::op_tp worker waiting -1733> 2014-05-19 18:38:07.424328 7fd04e104700 20 KeyValueStore::op_tp worker waiting -1732> 2014-05-19 18:38:07.424330 7fd0426ba700 20 OSD::op_tp worker waiting -1731> 2014-05-19 18:38:07.424368 7fd03b6ac700 20 OSD::op_tp worker waiting -1730> 2014-05-19 18:38:07.424387 7fd0416b8700 20 OSD::op_tp worker waiting -1729> 2014-05-19 18:38:07.424441 7fd03ceaf700 20 OSD::op_tp worker waiting -1728> 2014-05-19 18:38:07.502908 7fd04f907700 5 osd.0 21 tick -1727> 2014-05-19 18:38:07.502925 7fd04f907700 7 osd.0 21 do_mon_report -1726> 2014-05-19 18:38:07.502929 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -1725> 2014-05-19 18:38:07.502933 7fd04f907700 20 osd.0 21 send_pg_stats -1724> 2014-05-19 18:38:07.502968 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -1723> 2014-05-19 18:38:07.502980 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -1722> 2014-05-19 18:38:07.502985 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1721> 2014-05-19 18:38:07.502983 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 330.227 < min (86400 seconds) -1720> 2014-05-19 18:38:07.503002 7fd04f907700 20 osd.0 21 sched_scrub done -1719> 2014-05-19 18:38:07.503005 7fd04f907700 10 osd.0 21 do_waiters -- start -1718> 2014-05-19 18:38:07.503006 7fd04f907700 10 osd.0 21 do_waiters -- finish -1717> 2014-05-19 18:38:07.573415 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:10.631429) v2 ==== 47+0+0 (3456222985 0 0) 0x64d3180 con 0x4cb8dc0 -1716> 2014-05-19 18:38:07.573428 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:10.631429) v2 ==== 47+0+0 (3456222985 0 0) 0x5c468c0 con 0x4cb9340 -1715> 2014-05-19 18:38:07.573459 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:10.631429) v2 -- ?+0 0x656cfc0 con 0x4cb8dc0 -1714> 2014-05-19 18:38:07.573529 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1713> 2014-05-19 18:38:07.573554 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:10.631429) v2 -- ?+0 0x36f68c0 con 0x4cb9340 -1712> 2014-05-19 18:38:07.573633 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1711> 2014-05-19 18:38:08.503041 7fd04f907700 5 osd.0 21 tick -1710> 2014-05-19 18:38:08.503089 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -1709> 2014-05-19 18:38:08.503102 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -1708> 2014-05-19 18:38:08.503098 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1707> 2014-05-19 18:38:08.503105 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 331.227 < min (86400 seconds) -1706> 2014-05-19 18:38:08.503123 7fd04f907700 20 osd.0 21 sched_scrub done -1705> 2014-05-19 18:38:08.503126 7fd04f907700 10 osd.0 21 do_waiters -- start -1704> 2014-05-19 18:38:08.503128 7fd04f907700 10 osd.0 21 do_waiters -- finish -1703> 2014-05-19 18:38:08.778713 7fd04e905700 20 KeyValueStore::op_tp worker waiting -1702> 2014-05-19 18:38:08.778723 7fd043ebd700 20 OSD::op_tp worker waiting -1701> 2014-05-19 18:38:08.800568 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:08.800382) v2 ==== 47+0+0 (692798485 0 0) 0x3525a40 con 0x4cb8420 -1700> 2014-05-19 18:38:08.800612 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:08.800382) v2 ==== 47+0+0 (692798485 0 0) 0x36f6000 con 0x4cb86e0 -1699> 2014-05-19 18:38:08.800646 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:08.800382) v2 -- ?+0 0x5c468c0 con 0x4cb8420 -1698> 2014-05-19 18:38:08.800679 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1697> 2014-05-19 18:38:08.800706 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:08.800382) v2 -- ?+0 0x64d3180 con 0x4cb86e0 -1696> 2014-05-19 18:38:08.800802 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1695> 2014-05-19 18:38:08.950836 7fd0396a8700 20 OSD::command_tp worker waiting -1694> 2014-05-19 18:38:08.996571 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:14.086406) v2 ==== 47+0+0 (3210094682 0 0) 0x5deb500 con 0x5b21600 -1693> 2014-05-19 18:38:08.996635 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:14.086406) v2 -- ?+0 0x36f6000 con 0x5b21600 -1692> 2014-05-19 18:38:08.996667 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1691> 2014-05-19 18:38:08.996539 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:14.086406) v2 ==== 47+0+0 (3210094682 0 0) 0x70a9a40 con 0x5f5b760 -1690> 2014-05-19 18:38:08.996704 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:14.086406) v2 -- ?+0 0x3525a40 con 0x5f5b760 -1689> 2014-05-19 18:38:08.996736 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1688> 2014-05-19 18:38:09.345239 7fd03f6b4700 20 OSD::op_tp worker waiting -1687> 2014-05-19 18:38:09.345261 7fd03d6b0700 20 OSD::op_tp worker waiting -1686> 2014-05-19 18:38:09.345280 7fd0436bc700 20 OSD::op_tp worker waiting -1685> 2014-05-19 18:38:09.345300 7fd03bead700 20 OSD::op_tp worker waiting -1684> 2014-05-19 18:38:09.345320 7fd041eb9700 20 OSD::op_tp worker waiting -1683> 2014-05-19 18:38:09.345333 7fd03e6b2700 20 OSD::op_tp worker waiting -1682> 2014-05-19 18:38:09.345351 7fd03c6ae700 20 OSD::op_tp worker waiting -1681> 2014-05-19 18:38:09.359809 7fd0446be700 20 OSD::op_tp worker waiting -1680> 2014-05-19 18:38:09.359832 7fd03feb5700 20 OSD::op_tp worker waiting -1679> 2014-05-19 18:38:09.366987 7fd040eb7700 20 OSD::op_tp worker waiting -1678> 2014-05-19 18:38:09.372352 7fd0406b6700 20 OSD::op_tp worker waiting -1677> 2014-05-19 18:38:09.393943 7fd039ea9700 20 OSD::disk_tp worker waiting -1676> 2014-05-19 18:38:09.397486 7fd03aeab700 20 OSD::op_tp worker waiting -1675> 2014-05-19 18:38:09.397511 7fd03eeb3700 20 OSD::op_tp worker waiting -1674> 2014-05-19 18:38:09.415371 7fd042ebb700 20 OSD::op_tp worker waiting -1673> 2014-05-19 18:38:09.415391 7fd03deb1700 20 OSD::op_tp worker waiting -1672> 2014-05-19 18:38:09.424436 7fd04e104700 20 KeyValueStore::op_tp worker waiting -1671> 2014-05-19 18:38:09.424440 7fd0426ba700 20 OSD::op_tp worker waiting -1670> 2014-05-19 18:38:09.424477 7fd03b6ac700 20 OSD::op_tp worker waiting -1669> 2014-05-19 18:38:09.424495 7fd0416b8700 20 OSD::op_tp worker waiting -1668> 2014-05-19 18:38:09.424507 7fd03ceaf700 20 OSD::op_tp worker waiting -1667> 2014-05-19 18:38:09.503197 7fd04f907700 5 osd.0 21 tick -1666> 2014-05-19 18:38:09.503217 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1665> 2014-05-19 18:38:09.503222 7fd04f907700 10 osd.0 21 do_waiters -- start -1664> 2014-05-19 18:38:09.503224 7fd04f907700 10 osd.0 21 do_waiters -- finish -1663> 2014-05-19 18:38:09.503265 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1662> 2014-05-19 18:38:09.503837 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:12.561934) v2 ==== 47+0+0 (497409051 0 0) 0x7a37c00 con 0x61ae160 -1661> 2014-05-19 18:38:09.503868 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:12.561934) v2 ==== 47+0+0 (497409051 0 0) 0x64ae000 con 0x61b0420 -1660> 2014-05-19 18:38:09.503888 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:12.561934) v2 -- ?+0 0x5deb500 con 0x61ae160 -1659> 2014-05-19 18:38:09.503968 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1658> 2014-05-19 18:38:09.504012 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:12.561934) v2 -- ?+0 0x70a9a40 con 0x61b0420 -1657> 2014-05-19 18:38:09.504058 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1656> 2014-05-19 18:38:09.670999 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -1655> 2014-05-19 18:38:09.671017 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -1654> 2014-05-19 18:38:09.671025 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x77e96c0 con 0x3622580 -1653> 2014-05-19 18:38:09.671051 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x59bc1c0 con 0x3622840 -1652> 2014-05-19 18:38:09.671090 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x6c4b180 con 0x3622b00 -1651> 2014-05-19 18:38:09.671104 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x6009dc0 con 0x3622dc0 -1650> 2014-05-19 18:38:09.671134 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x77e81c0 con 0x30594a0 -1649> 2014-05-19 18:38:09.671159 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x7999dc0 con 0x3622f20 -1648> 2014-05-19 18:38:09.671190 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x8388700 con 0x5e7ef20 -1647> 2014-05-19 18:38:09.671215 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x691b6c0 con 0x5e7f1e0 -1646> 2014-05-19 18:38:09.671241 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x711ee00 con 0x5e7e2c0 -1645> 2014-05-19 18:38:09.671266 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x67101c0 con 0x5d7bb80 -1644> 2014-05-19 18:38:09.671288 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x66f9a40 con 0x5e7ec60 -1643> 2014-05-19 18:38:09.671312 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x798e1c0 con 0x531b760 -1642> 2014-05-19 18:38:09.671338 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x7980380 con 0x53bba20 -1641> 2014-05-19 18:38:09.671363 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x5cbc540 con 0x51cac60 -1640> 2014-05-19 18:38:09.671389 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x3525880 con 0x5bd66e0 -1639> 2014-05-19 18:38:09.671413 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:09.671024) v2 -- ?+0 0x5e28380 con 0x5bd6840 -1638> 2014-05-19 18:38:09.671404 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x8388a80 con 0x3622dc0 -1637> 2014-05-19 18:38:09.671471 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1636> 2014-05-19 18:38:09.671507 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x6710fc0 con 0x3622580 -1635> 2014-05-19 18:38:09.671558 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1634> 2014-05-19 18:38:09.671585 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x72e4540 con 0x3622b00 -1633> 2014-05-19 18:38:09.671611 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1632> 2014-05-19 18:38:09.671620 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x5667c00 con 0x30594a0 -1631> 2014-05-19 18:38:09.671636 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1630> 2014-05-19 18:38:09.671641 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x682e700 con 0x3622f20 -1629> 2014-05-19 18:38:09.671669 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1628> 2014-05-19 18:38:09.671702 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x7a5e8c0 con 0x3622840 -1627> 2014-05-19 18:38:09.671738 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1626> 2014-05-19 18:38:09.671827 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x51a68c0 con 0x5e7ef20 -1625> 2014-05-19 18:38:09.671856 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1624> 2014-05-19 18:38:09.671863 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x60088c0 con 0x5e7f1e0 -1623> 2014-05-19 18:38:09.671878 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1622> 2014-05-19 18:38:09.671883 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x5f0a380 con 0x5e7e2c0 -1621> 2014-05-19 18:38:09.671897 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1620> 2014-05-19 18:38:09.671903 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x8648e00 con 0x5d7bb80 -1619> 2014-05-19 18:38:09.671921 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1618> 2014-05-19 18:38:09.671931 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x798e380 con 0x5e7ec60 -1617> 2014-05-19 18:38:09.671946 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1616> 2014-05-19 18:38:09.671953 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x2ee16c0 con 0x531b760 -1615> 2014-05-19 18:38:09.671968 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1614> 2014-05-19 18:38:09.671975 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x6c4c540 con 0x53bba20 -1613> 2014-05-19 18:38:09.671991 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1612> 2014-05-19 18:38:09.671997 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x5e28e00 con 0x51cac60 -1611> 2014-05-19 18:38:09.672013 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1610> 2014-05-19 18:38:09.672020 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x70a9340 con 0x5bd66e0 -1609> 2014-05-19 18:38:09.672035 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1608> 2014-05-19 18:38:09.672043 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 96 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:09.671024) v2 ==== 47+0+0 (3577315541 0 0) 0x6112e00 con 0x5bd6840 -1607> 2014-05-19 18:38:09.672060 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1606> 2014-05-19 18:38:09.874101 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:12.932120) v2 ==== 47+0+0 (500905407 0 0) 0x77341c0 con 0x4cb9340 -1605> 2014-05-19 18:38:09.874163 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:12.932120) v2 -- ?+0 0x64ae000 con 0x4cb9340 -1604> 2014-05-19 18:38:09.874242 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1603> 2014-05-19 18:38:09.874278 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:12.932120) v2 ==== 47+0+0 (500905407 0 0) 0x6543180 con 0x4cb8dc0 -1602> 2014-05-19 18:38:09.874306 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:12.932120) v2 -- ?+0 0x7a37c00 con 0x4cb8dc0 -1601> 2014-05-19 18:38:09.874326 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1600> 2014-05-19 18:38:09.908360 7fd0456c0700 10 monclient: tick -1599> 2014-05-19 18:38:09.908373 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:37:39.908371) -1598> 2014-05-19 18:38:09.908389 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:38:09.908389; renew after: 2014-05-19 18:38:19.906664) -- no -1597> 2014-05-19 18:38:09.921334 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:15.011215) v2 ==== 47+0+0 (2112511262 0 0) 0x79e6a80 con 0x30946e0 -1596> 2014-05-19 18:38:09.921379 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 91 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:15.011215) v2 ==== 47+0+0 (2112511262 0 0) 0x55c36c0 con 0x3094160 -1595> 2014-05-19 18:38:09.921387 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:15.011215) v2 -- ?+0 0x6543180 con 0x30946e0 -1594> 2014-05-19 18:38:09.921469 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1593> 2014-05-19 18:38:09.921496 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:15.011215) v2 -- ?+0 0x77341c0 con 0x3094160 -1592> 2014-05-19 18:38:09.921528 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1591> 2014-05-19 18:38:10.382790 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:15.472699) v2 ==== 47+0+0 (1390318922 0 0) 0x78b9a40 con 0x61b0160 -1590> 2014-05-19 18:38:10.382843 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:15.472699) v2 -- ?+0 0x55c36c0 con 0x61b0160 -1589> 2014-05-19 18:38:10.382875 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1588> 2014-05-19 18:38:10.382790 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:15.472699) v2 ==== 47+0+0 (1390318922 0 0) 0x60a1a40 con 0x31e1600 -1587> 2014-05-19 18:38:10.382916 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:15.472699) v2 -- ?+0 0x79e6a80 con 0x31e1600 -1586> 2014-05-19 18:38:10.382935 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1585> 2014-05-19 18:38:10.503292 7fd04f907700 5 osd.0 21 tick -1584> 2014-05-19 18:38:10.503314 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1583> 2014-05-19 18:38:10.503319 7fd04f907700 10 osd.0 21 do_waiters -- start -1582> 2014-05-19 18:38:10.503321 7fd04f907700 10 osd.0 21 do_waiters -- finish -1581> 2014-05-19 18:38:10.503362 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1580> 2014-05-19 18:38:10.778798 7fd04e905700 20 KeyValueStore::op_tp worker waiting -1579> 2014-05-19 18:38:10.778800 7fd043ebd700 20 OSD::op_tp worker waiting -1578> 2014-05-19 18:38:10.950926 7fd0396a8700 20 OSD::command_tp worker waiting -1577> 2014-05-19 18:38:11.345336 7fd03f6b4700 20 OSD::op_tp worker waiting -1576> 2014-05-19 18:38:11.345355 7fd03bead700 20 OSD::op_tp worker waiting -1575> 2014-05-19 18:38:11.345374 7fd0436bc700 20 OSD::op_tp worker waiting -1574> 2014-05-19 18:38:11.345395 7fd03d6b0700 20 OSD::op_tp worker waiting -1573> 2014-05-19 18:38:11.345409 7fd03e6b2700 20 OSD::op_tp worker waiting -1572> 2014-05-19 18:38:11.345419 7fd041eb9700 20 OSD::op_tp worker waiting -1571> 2014-05-19 18:38:11.345442 7fd03c6ae700 20 OSD::op_tp worker waiting -1570> 2014-05-19 18:38:11.359885 7fd0446be700 20 OSD::op_tp worker waiting -1569> 2014-05-19 18:38:11.359904 7fd03feb5700 20 OSD::op_tp worker waiting -1568> 2014-05-19 18:38:11.367084 7fd040eb7700 20 OSD::op_tp worker waiting -1567> 2014-05-19 18:38:11.372448 7fd0406b6700 20 OSD::op_tp worker waiting -1566> 2014-05-19 18:38:11.394040 7fd039ea9700 20 OSD::disk_tp worker waiting -1565> 2014-05-19 18:38:11.397587 7fd03aeab700 20 OSD::op_tp worker waiting -1564> 2014-05-19 18:38:11.397615 7fd03eeb3700 20 OSD::op_tp worker waiting -1563> 2014-05-19 18:38:11.415445 7fd042ebb700 20 OSD::op_tp worker waiting -1562> 2014-05-19 18:38:11.415468 7fd03deb1700 20 OSD::op_tp worker waiting -1561> 2014-05-19 18:38:11.424548 7fd04e104700 20 KeyValueStore::op_tp worker waiting -1560> 2014-05-19 18:38:11.424551 7fd03b6ac700 20 OSD::op_tp worker waiting -1559> 2014-05-19 18:38:11.424594 7fd0426ba700 20 OSD::op_tp worker waiting -1558> 2014-05-19 18:38:11.424620 7fd03ceaf700 20 OSD::op_tp worker waiting -1557> 2014-05-19 18:38:11.424673 7fd0416b8700 20 OSD::op_tp worker waiting -1556> 2014-05-19 18:38:11.435032 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:14.493086) v2 ==== 47+0+0 (108800876 0 0) 0x5b281c0 con 0x618fe40 -1555> 2014-05-19 18:38:11.435093 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:14.493086) v2 -- ?+0 0x78b9a40 con 0x618fe40 -1554> 2014-05-19 18:38:11.435132 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1553> 2014-05-19 18:38:11.435047 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:14.493086) v2 ==== 47+0+0 (108800876 0 0) 0x66f8540 con 0x61ae420 -1552> 2014-05-19 18:38:11.435151 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:14.493086) v2 -- ?+0 0x60a1a40 con 0x61ae420 -1551> 2014-05-19 18:38:11.435239 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1550> 2014-05-19 18:38:11.503392 7fd04f907700 5 osd.0 21 tick -1549> 2014-05-19 18:38:11.503413 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1548> 2014-05-19 18:38:11.503418 7fd04f907700 10 osd.0 21 do_waiters -- start -1547> 2014-05-19 18:38:11.503420 7fd04f907700 10 osd.0 21 do_waiters -- finish -1546> 2014-05-19 18:38:11.503443 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1545> 2014-05-19 18:38:11.586672 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:11.586480) v2 ==== 47+0+0 (67823698 0 0) 0x656c000 con 0x4cb8160 -1544> 2014-05-19 18:38:11.586723 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:11.586480) v2 -- ?+0 0x66f8540 con 0x4cb8160 -1543> 2014-05-19 18:38:11.586735 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:11.586480) v2 ==== 47+0+0 (67823698 0 0) 0x36f7340 con 0x4cb82c0 -1542> 2014-05-19 18:38:11.586775 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1541> 2014-05-19 18:38:11.586795 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:11.586480) v2 -- ?+0 0x5b281c0 con 0x4cb82c0 -1540> 2014-05-19 18:38:11.586835 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1539> 2014-05-19 18:38:11.804385 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:14.862446) v2 ==== 47+0+0 (139269740 0 0) 0x5c47880 con 0x61ae160 -1538> 2014-05-19 18:38:11.804401 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:14.862446) v2 ==== 47+0+0 (139269740 0 0) 0x799a700 con 0x61b0420 -1537> 2014-05-19 18:38:11.804435 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:14.862446) v2 -- ?+0 0x656c000 con 0x61ae160 -1536> 2014-05-19 18:38:11.804517 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1535> 2014-05-19 18:38:11.804545 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:14.862446) v2 -- ?+0 0x36f7340 con 0x61b0420 -1534> 2014-05-19 18:38:11.804605 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1533> 2014-05-19 18:38:12.503453 7fd04f907700 5 osd.0 21 tick -1532> 2014-05-19 18:38:12.503469 7fd04f907700 7 osd.0 21 do_mon_report -1531> 2014-05-19 18:38:12.503473 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -1530> 2014-05-19 18:38:12.503477 7fd04f907700 20 osd.0 21 send_pg_stats -1529> 2014-05-19 18:38:12.503514 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -1528> 2014-05-19 18:38:12.503526 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -1527> 2014-05-19 18:38:12.503530 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1526> 2014-05-19 18:38:12.503529 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 335.228 < min (86400 seconds) -1525> 2014-05-19 18:38:12.503548 7fd04f907700 20 osd.0 21 sched_scrub done -1524> 2014-05-19 18:38:12.503551 7fd04f907700 10 osd.0 21 do_waiters -- start -1523> 2014-05-19 18:38:12.503553 7fd04f907700 10 osd.0 21 do_waiters -- finish -1522> 2014-05-19 18:38:12.683243 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:17.773214) v2 ==== 47+0+0 (1652645188 0 0) 0x7a37180 con 0x31e1600 -1521> 2014-05-19 18:38:12.683257 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:17.773214) v2 ==== 47+0+0 (1652645188 0 0) 0x78b8c40 con 0x61b0160 -1520> 2014-05-19 18:38:12.683293 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.773214) v2 -- ?+0 0x5c47880 con 0x31e1600 -1519> 2014-05-19 18:38:12.683338 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1518> 2014-05-19 18:38:12.683394 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.773214) v2 -- ?+0 0x799a700 con 0x61b0160 -1517> 2014-05-19 18:38:12.683445 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1516> 2014-05-19 18:38:12.778899 7fd04e905700 20 KeyValueStore::op_tp worker waiting -1515> 2014-05-19 18:38:12.778902 7fd043ebd700 20 OSD::op_tp worker waiting -1514> 2014-05-19 18:38:12.951021 7fd0396a8700 20 OSD::command_tp worker waiting -1513> 2014-05-19 18:38:13.345421 7fd03f6b4700 20 OSD::op_tp worker waiting -1512> 2014-05-19 18:38:13.345447 7fd03bead700 20 OSD::op_tp worker waiting -1511> 2014-05-19 18:38:13.345469 7fd0436bc700 20 OSD::op_tp worker waiting -1510> 2014-05-19 18:38:13.345491 7fd041eb9700 20 OSD::op_tp worker waiting -1509> 2014-05-19 18:38:13.345507 7fd03d6b0700 20 OSD::op_tp worker waiting -1508> 2014-05-19 18:38:13.345526 7fd03e6b2700 20 OSD::op_tp worker waiting -1507> 2014-05-19 18:38:13.345553 7fd03c6ae700 20 OSD::op_tp worker waiting -1506> 2014-05-19 18:38:13.359961 7fd0446be700 20 OSD::op_tp worker waiting -1505> 2014-05-19 18:38:13.359989 7fd03feb5700 20 OSD::op_tp worker waiting -1504> 2014-05-19 18:38:13.367167 7fd040eb7700 20 OSD::op_tp worker waiting -1503> 2014-05-19 18:38:13.372543 7fd0406b6700 20 OSD::op_tp worker waiting -1502> 2014-05-19 18:38:13.394120 7fd039ea9700 20 OSD::disk_tp worker waiting -1501> 2014-05-19 18:38:13.397687 7fd03aeab700 20 OSD::op_tp worker waiting -1500> 2014-05-19 18:38:13.397713 7fd03eeb3700 20 OSD::op_tp worker waiting -1499> 2014-05-19 18:38:13.415515 7fd042ebb700 20 OSD::op_tp worker waiting -1498> 2014-05-19 18:38:13.415546 7fd03deb1700 20 OSD::op_tp worker waiting -1497> 2014-05-19 18:38:13.424644 7fd04e104700 20 KeyValueStore::op_tp worker waiting -1496> 2014-05-19 18:38:13.424649 7fd03b6ac700 20 OSD::op_tp worker waiting -1495> 2014-05-19 18:38:13.424673 7fd0426ba700 20 OSD::op_tp worker waiting -1494> 2014-05-19 18:38:13.424727 7fd03ceaf700 20 OSD::op_tp worker waiting -1493> 2014-05-19 18:38:13.424757 7fd0416b8700 20 OSD::op_tp worker waiting -1492> 2014-05-19 18:38:13.501252 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:13.501072) v2 ==== 47+0+0 (609515276 0 0) 0x725ca80 con 0x4cb86e0 -1491> 2014-05-19 18:38:13.501319 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.501072) v2 -- ?+0 0x7a37180 con 0x4cb86e0 -1490> 2014-05-19 18:38:13.501356 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1489> 2014-05-19 18:38:13.501276 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:13.501072) v2 ==== 47+0+0 (609515276 0 0) 0x656c8c0 con 0x4cb8420 -1488> 2014-05-19 18:38:13.501387 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.501072) v2 -- ?+0 0x78b8c40 con 0x4cb8420 -1487> 2014-05-19 18:38:13.501422 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1486> 2014-05-19 18:38:13.503622 7fd04f907700 5 osd.0 21 tick -1485> 2014-05-19 18:38:13.503649 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1484> 2014-05-19 18:38:13.503672 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -1483> 2014-05-19 18:38:13.503685 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -1482> 2014-05-19 18:38:13.503688 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 336.228 < min (86400 seconds) -1481> 2014-05-19 18:38:13.503708 7fd04f907700 20 osd.0 21 sched_scrub done -1480> 2014-05-19 18:38:13.503711 7fd04f907700 10 osd.0 21 do_waiters -- start -1479> 2014-05-19 18:38:13.503713 7fd04f907700 10 osd.0 21 do_waiters -- finish -1478> 2014-05-19 18:38:13.771560 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -1477> 2014-05-19 18:38:13.771578 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -1476> 2014-05-19 18:38:13.771586 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x6c4ca80 con 0x3622580 -1475> 2014-05-19 18:38:13.771611 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x77e8c40 con 0x3622840 -1474> 2014-05-19 18:38:13.771649 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x7746c40 con 0x3622b00 -1473> 2014-05-19 18:38:13.771663 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x59bc540 con 0x3622dc0 -1472> 2014-05-19 18:38:13.771676 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x7733dc0 con 0x30594a0 -1471> 2014-05-19 18:38:13.771696 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x5ccc8c0 con 0x3622f20 -1470> 2014-05-19 18:38:13.771716 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x711fa40 con 0x5e7ef20 -1469> 2014-05-19 18:38:13.771734 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x5f328c0 con 0x5e7f1e0 -1468> 2014-05-19 18:38:13.771748 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x72e5500 con 0x5e7e2c0 -1467> 2014-05-19 18:38:13.771763 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x6504700 con 0x5d7bb80 -1466> 2014-05-19 18:38:13.771793 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x662b180 con 0x5e7ec60 -1465> 2014-05-19 18:38:13.771821 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x5cbcfc0 con 0x531b760 -1464> 2014-05-19 18:38:13.771846 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x3524380 con 0x53bba20 -1463> 2014-05-19 18:38:13.771870 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x5496000 con 0x51cac60 -1462> 2014-05-19 18:38:13.771895 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x6c4c700 con 0x5bd66e0 -1461> 2014-05-19 18:38:13.771919 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:13.771584) v2 -- ?+0 0x66f88c0 con 0x5bd6840 -1460> 2014-05-19 18:38:13.771985 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x6711500 con 0x3622580 -1459> 2014-05-19 18:38:13.772047 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1458> 2014-05-19 18:38:13.772070 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x72e4700 con 0x3622b00 -1457> 2014-05-19 18:38:13.772091 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1456> 2014-05-19 18:38:13.772101 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x8389880 con 0x3622dc0 -1455> 2014-05-19 18:38:13.772116 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1454> 2014-05-19 18:38:13.772121 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x7a5ea80 con 0x3622840 -1453> 2014-05-19 18:38:13.772241 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1452> 2014-05-19 18:38:13.772280 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x682fc00 con 0x3622f20 -1451> 2014-05-19 18:38:13.772312 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1450> 2014-05-19 18:38:13.772330 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x73c21c0 con 0x5e7ef20 -1449> 2014-05-19 18:38:13.772363 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1448> 2014-05-19 18:38:13.772378 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x6009180 con 0x5e7f1e0 -1447> 2014-05-19 18:38:13.772400 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1446> 2014-05-19 18:38:13.772408 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x5f0b180 con 0x5e7e2c0 -1445> 2014-05-19 18:38:13.772425 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1444> 2014-05-19 18:38:13.772433 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x8649880 con 0x5d7bb80 -1443> 2014-05-19 18:38:13.772450 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1442> 2014-05-19 18:38:13.772458 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x798e540 con 0x5e7ec60 -1441> 2014-05-19 18:38:13.772474 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1440> 2014-05-19 18:38:13.772484 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x2ee1500 con 0x531b760 -1439> 2014-05-19 18:38:13.772499 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1438> 2014-05-19 18:38:13.772505 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x6c4d6c0 con 0x53bba20 -1437> 2014-05-19 18:38:13.772522 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1436> 2014-05-19 18:38:13.772532 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x5e288c0 con 0x51cac60 -1435> 2014-05-19 18:38:13.772548 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1434> 2014-05-19 18:38:13.772556 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x70a9dc0 con 0x5bd66e0 -1433> 2014-05-19 18:38:13.772593 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1432> 2014-05-19 18:38:13.772603 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 97 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x6113180 con 0x5bd6840 -1431> 2014-05-19 18:38:13.772619 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1430> 2014-05-19 18:38:13.772625 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:13.771584) v2 ==== 47+0+0 (1452152165 0 0) 0x5667340 con 0x30594a0 -1429> 2014-05-19 18:38:13.772639 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1428> 2014-05-19 18:38:14.503797 7fd04f907700 5 osd.0 21 tick -1427> 2014-05-19 18:38:14.503846 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -1426> 2014-05-19 18:38:14.503859 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -1425> 2014-05-19 18:38:14.503866 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1424> 2014-05-19 18:38:14.503862 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 337.228 < min (86400 seconds) -1423> 2014-05-19 18:38:14.503881 7fd04f907700 20 osd.0 21 sched_scrub done -1422> 2014-05-19 18:38:14.503883 7fd04f907700 10 osd.0 21 do_waiters -- start -1421> 2014-05-19 18:38:14.503885 7fd04f907700 10 osd.0 21 do_waiters -- finish -1420> 2014-05-19 18:38:14.574597 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:17.632688) v2 ==== 47+0+0 (2611696089 0 0) 0x7734000 con 0x4cb9340 -1419> 2014-05-19 18:38:14.574609 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:17.632688) v2 ==== 47+0+0 (2611696089 0 0) 0x6542c40 con 0x4cb8dc0 -1418> 2014-05-19 18:38:14.574658 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.632688) v2 -- ?+0 0x656c8c0 con 0x4cb9340 -1417> 2014-05-19 18:38:14.574685 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1416> 2014-05-19 18:38:14.574710 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.632688) v2 -- ?+0 0x725ca80 con 0x4cb8dc0 -1415> 2014-05-19 18:38:14.574750 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1414> 2014-05-19 18:38:14.778992 7fd04e905700 20 KeyValueStore::op_tp worker waiting -1413> 2014-05-19 18:38:14.778994 7fd043ebd700 20 OSD::op_tp worker waiting -1412> 2014-05-19 18:38:14.896832 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:19.986873) v2 ==== 47+0+0 (1529608018 0 0) 0x5debc00 con 0x5b21600 -1411> 2014-05-19 18:38:14.896842 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:19.986873) v2 ==== 47+0+0 (1529608018 0 0) 0x76ac1c0 con 0x5f5b760 -1410> 2014-05-19 18:38:14.896914 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:19.986873) v2 -- ?+0 0x6542c40 con 0x5b21600 -1409> 2014-05-19 18:38:14.896944 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1408> 2014-05-19 18:38:14.896964 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:19.986873) v2 -- ?+0 0x7734000 con 0x5f5b760 -1407> 2014-05-19 18:38:14.897002 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1406> 2014-05-19 18:38:14.951102 7fd0396a8700 20 OSD::command_tp worker waiting -1405> 2014-05-19 18:38:15.345513 7fd03bead700 20 OSD::op_tp worker waiting -1404> 2014-05-19 18:38:15.345548 7fd041eb9700 20 OSD::op_tp worker waiting -1403> 2014-05-19 18:38:15.345566 7fd0436bc700 20 OSD::op_tp worker waiting -1402> 2014-05-19 18:38:15.345583 7fd03f6b4700 20 OSD::op_tp worker waiting -1401> 2014-05-19 18:38:15.345601 7fd03d6b0700 20 OSD::op_tp worker waiting -1400> 2014-05-19 18:38:15.345632 7fd03e6b2700 20 OSD::op_tp worker waiting -1399> 2014-05-19 18:38:15.345684 7fd03c6ae700 20 OSD::op_tp worker waiting -1398> 2014-05-19 18:38:15.360037 7fd0446be700 20 OSD::op_tp worker waiting -1397> 2014-05-19 18:38:15.360095 7fd03feb5700 20 OSD::op_tp worker waiting -1396> 2014-05-19 18:38:15.367250 7fd040eb7700 20 OSD::op_tp worker waiting -1395> 2014-05-19 18:38:15.372639 7fd0406b6700 20 OSD::op_tp worker waiting -1394> 2014-05-19 18:38:15.394199 7fd039ea9700 20 OSD::disk_tp worker waiting -1393> 2014-05-19 18:38:15.397787 7fd03aeab700 20 OSD::op_tp worker waiting -1392> 2014-05-19 18:38:15.397811 7fd03eeb3700 20 OSD::op_tp worker waiting -1391> 2014-05-19 18:38:15.415596 7fd042ebb700 20 OSD::op_tp worker waiting -1390> 2014-05-19 18:38:15.415618 7fd03deb1700 20 OSD::op_tp worker waiting -1389> 2014-05-19 18:38:15.424740 7fd04e104700 20 KeyValueStore::op_tp worker waiting -1388> 2014-05-19 18:38:15.424745 7fd03b6ac700 20 OSD::op_tp worker waiting -1387> 2014-05-19 18:38:15.424766 7fd0426ba700 20 OSD::op_tp worker waiting -1386> 2014-05-19 18:38:15.424793 7fd03ceaf700 20 OSD::op_tp worker waiting -1385> 2014-05-19 18:38:15.424827 7fd0416b8700 20 OSD::op_tp worker waiting -1384> 2014-05-19 18:38:15.503970 7fd04f907700 5 osd.0 21 tick -1383> 2014-05-19 18:38:15.503990 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1382> 2014-05-19 18:38:15.503995 7fd04f907700 10 osd.0 21 do_waiters -- start -1381> 2014-05-19 18:38:15.503998 7fd04f907700 10 osd.0 21 do_waiters -- finish -1380> 2014-05-19 18:38:15.504038 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1379> 2014-05-19 18:38:15.535594 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:18.593748) v2 ==== 47+0+0 (725277716 0 0) 0x64afdc0 con 0x618fe40 -1378> 2014-05-19 18:38:15.535605 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:18.593748) v2 ==== 47+0+0 (725277716 0 0) 0x70cda40 con 0x61ae420 -1377> 2014-05-19 18:38:15.535646 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:18.593748) v2 -- ?+0 0x76ac1c0 con 0x618fe40 -1376> 2014-05-19 18:38:15.535681 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1375> 2014-05-19 18:38:15.535702 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:18.593748) v2 -- ?+0 0x5debc00 con 0x61ae420 -1374> 2014-05-19 18:38:15.535744 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1373> 2014-05-19 18:38:15.687192 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:15.686999) v2 ==== 47+0+0 (2765455944 0 0) 0x656ce00 con 0x4cb8160 -1372> 2014-05-19 18:38:15.687220 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:15.686999) v2 ==== 47+0+0 (2765455944 0 0) 0x60e2000 con 0x4cb82c0 -1371> 2014-05-19 18:38:15.687246 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:15.686999) v2 -- ?+0 0x70cda40 con 0x4cb8160 -1370> 2014-05-19 18:38:15.687271 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1369> 2014-05-19 18:38:15.687499 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:15.686999) v2 -- ?+0 0x64afdc0 con 0x4cb82c0 -1368> 2014-05-19 18:38:15.687553 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1367> 2014-05-19 18:38:15.821585 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:20.911687) v2 ==== 47+0+0 (4021886749 0 0) 0x34348c0 con 0x3094160 -1366> 2014-05-19 18:38:15.821589 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 92 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:20.911687) v2 ==== 47+0+0 (4021886749 0 0) 0x7570000 con 0x30946e0 -1365> 2014-05-19 18:38:15.821642 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.911687) v2 -- ?+0 0x60e2000 con 0x3094160 -1364> 2014-05-19 18:38:15.821675 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1363> 2014-05-19 18:38:15.821700 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.911687) v2 -- ?+0 0x656ce00 con 0x30946e0 -1362> 2014-05-19 18:38:15.821738 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1361> 2014-05-19 18:38:16.504051 7fd04f907700 5 osd.0 21 tick -1360> 2014-05-19 18:38:16.504104 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -1359> 2014-05-19 18:38:16.504118 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -1358> 2014-05-19 18:38:16.504120 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1357> 2014-05-19 18:38:16.504121 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 339.228 < min (86400 seconds) -1356> 2014-05-19 18:38:16.504139 7fd04f907700 20 osd.0 21 sched_scrub done -1355> 2014-05-19 18:38:16.504142 7fd04f907700 10 osd.0 21 do_waiters -- start -1354> 2014-05-19 18:38:16.504144 7fd04f907700 10 osd.0 21 do_waiters -- finish -1353> 2014-05-19 18:38:16.504800 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:19.562975) v2 ==== 47+0+0 (1339789787 0 0) 0x5519500 con 0x61ae160 -1352> 2014-05-19 18:38:16.504852 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:19.562975) v2 -- ?+0 0x7570000 con 0x61ae160 -1351> 2014-05-19 18:38:16.504864 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:19.562975) v2 ==== 47+0+0 (1339789787 0 0) 0x66f81c0 con 0x61b0420 -1350> 2014-05-19 18:38:16.504936 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1349> 2014-05-19 18:38:16.504977 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:19.562975) v2 -- ?+0 0x34348c0 con 0x61b0420 -1348> 2014-05-19 18:38:16.505017 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1347> 2014-05-19 18:38:16.779096 7fd04e905700 20 KeyValueStore::op_tp worker waiting -1346> 2014-05-19 18:38:16.779098 7fd043ebd700 20 OSD::op_tp worker waiting -1345> 2014-05-19 18:38:16.951180 7fd0396a8700 20 OSD::command_tp worker waiting -1344> 2014-05-19 18:38:17.197228 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:22.287397) v2 ==== 47+0+0 (4101208640 0 0) 0x55c88c0 con 0x5f5b760 -1343> 2014-05-19 18:38:17.197252 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:22.287397) v2 ==== 47+0+0 (4101208640 0 0) 0x725da40 con 0x5b21600 -1342> 2014-05-19 18:38:17.197279 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:22.287397) v2 -- ?+0 0x66f81c0 con 0x5f5b760 -1341> 2014-05-19 18:38:17.197315 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1340> 2014-05-19 18:38:17.197335 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:22.287397) v2 -- ?+0 0x5519500 con 0x5b21600 -1339> 2014-05-19 18:38:17.197422 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1338> 2014-05-19 18:38:17.345605 7fd03bead700 20 OSD::op_tp worker waiting -1337> 2014-05-19 18:38:17.345627 7fd0436bc700 20 OSD::op_tp worker waiting -1336> 2014-05-19 18:38:17.345649 7fd041eb9700 20 OSD::op_tp worker waiting -1335> 2014-05-19 18:38:17.345673 7fd03f6b4700 20 OSD::op_tp worker waiting -1334> 2014-05-19 18:38:17.345693 7fd03d6b0700 20 OSD::op_tp worker waiting -1333> 2014-05-19 18:38:17.345708 7fd03e6b2700 20 OSD::op_tp worker waiting -1332> 2014-05-19 18:38:17.345754 7fd03c6ae700 20 OSD::op_tp worker waiting -1331> 2014-05-19 18:38:17.360111 7fd0446be700 20 OSD::op_tp worker waiting -1330> 2014-05-19 18:38:17.360174 7fd03feb5700 20 OSD::op_tp worker waiting -1329> 2014-05-19 18:38:17.367335 7fd040eb7700 20 OSD::op_tp worker waiting -1328> 2014-05-19 18:38:17.372720 7fd0406b6700 20 OSD::op_tp worker waiting -1327> 2014-05-19 18:38:17.383553 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:22.473742) v2 ==== 47+0+0 (943849229 0 0) 0x74a6700 con 0x31e1600 -1326> 2014-05-19 18:38:17.383603 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:22.473742) v2 -- ?+0 0x725da40 con 0x31e1600 -1325> 2014-05-19 18:38:17.383632 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1324> 2014-05-19 18:38:17.383757 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:22.473742) v2 ==== 47+0+0 (943849229 0 0) 0x78b9500 con 0x61b0160 -1323> 2014-05-19 18:38:17.383807 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:22.473742) v2 -- ?+0 0x55c88c0 con 0x61b0160 -1322> 2014-05-19 18:38:17.383836 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1321> 2014-05-19 18:38:17.394283 7fd039ea9700 20 OSD::disk_tp worker waiting -1320> 2014-05-19 18:38:17.397889 7fd03aeab700 20 OSD::op_tp worker waiting -1319> 2014-05-19 18:38:17.397947 7fd03eeb3700 20 OSD::op_tp worker waiting -1318> 2014-05-19 18:38:17.415677 7fd042ebb700 20 OSD::op_tp worker waiting -1317> 2014-05-19 18:38:17.415698 7fd03deb1700 20 OSD::op_tp worker waiting -1316> 2014-05-19 18:38:17.424842 7fd04e104700 20 KeyValueStore::op_tp worker waiting -1315> 2014-05-19 18:38:17.424843 7fd0426ba700 20 OSD::op_tp worker waiting -1314> 2014-05-19 18:38:17.424879 7fd03b6ac700 20 OSD::op_tp worker waiting -1313> 2014-05-19 18:38:17.424894 7fd0416b8700 20 OSD::op_tp worker waiting -1312> 2014-05-19 18:38:17.424910 7fd03ceaf700 20 OSD::op_tp worker waiting -1311> 2014-05-19 18:38:17.504230 7fd04f907700 5 osd.0 21 tick -1310> 2014-05-19 18:38:17.504247 7fd04f907700 7 osd.0 21 do_mon_report -1309> 2014-05-19 18:38:17.504251 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -1308> 2014-05-19 18:38:17.504254 7fd04f907700 20 osd.0 21 send_pg_stats -1307> 2014-05-19 18:38:17.504289 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -1306> 2014-05-19 18:38:17.504301 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -1305> 2014-05-19 18:38:17.504307 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1304> 2014-05-19 18:38:17.504304 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 340.228 < min (86400 seconds) -1303> 2014-05-19 18:38:17.504324 7fd04f907700 20 osd.0 21 sched_scrub done -1302> 2014-05-19 18:38:17.504327 7fd04f907700 10 osd.0 21 do_waiters -- start -1301> 2014-05-19 18:38:17.504328 7fd04f907700 10 osd.0 21 do_waiters -- finish -1300> 2014-05-19 18:38:17.601832 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:17.601611) v2 ==== 47+0+0 (3592019690 0 0) 0x725d880 con 0x4cb86e0 -1299> 2014-05-19 18:38:17.601882 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.601611) v2 -- ?+0 0x74a6700 con 0x4cb86e0 -1298> 2014-05-19 18:38:17.601920 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1297> 2014-05-19 18:38:17.601887 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:17.601611) v2 ==== 47+0+0 (3592019690 0 0) 0x656cc40 con 0x4cb8420 -1296> 2014-05-19 18:38:17.601994 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.601611) v2 -- ?+0 0x78b9500 con 0x4cb8420 -1295> 2014-05-19 18:38:17.602032 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1294> 2014-05-19 18:38:17.872065 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -1293> 2014-05-19 18:38:17.872083 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -1292> 2014-05-19 18:38:17.872091 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x77e8380 con 0x3622580 -1291> 2014-05-19 18:38:17.872116 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x77476c0 con 0x3622840 -1290> 2014-05-19 18:38:17.872153 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x6009c00 con 0x3622b00 -1289> 2014-05-19 18:38:17.872168 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x5f32380 con 0x3622dc0 -1288> 2014-05-19 18:38:17.872183 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x7a5e8c0 con 0x30594a0 -1287> 2014-05-19 18:38:17.872212 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x682e700 con 0x3622f20 -1286> 2014-05-19 18:38:17.872243 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x5667c00 con 0x5e7ef20 -1285> 2014-05-19 18:38:17.872269 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x72e4540 con 0x5e7f1e0 -1284> 2014-05-19 18:38:17.872295 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x6710fc0 con 0x5e7e2c0 -1283> 2014-05-19 18:38:17.872323 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x8388a80 con 0x5d7bb80 -1282> 2014-05-19 18:38:17.872350 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x70a8540 con 0x5e7ec60 -1281> 2014-05-19 18:38:17.872380 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x798fdc0 con 0x531b760 -1280> 2014-05-19 18:38:17.872409 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x7998a80 con 0x53bba20 -1279> 2014-05-19 18:38:17.872434 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x6113880 con 0x51cac60 -1278> 2014-05-19 18:38:17.872460 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x5e29340 con 0x5bd66e0 -1277> 2014-05-19 18:38:17.872495 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:17.872089) v2 -- ?+0 0x6c4cc40 con 0x5bd6840 -1276> 2014-05-19 18:38:17.872499 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x6710e00 con 0x3622580 -1275> 2014-05-19 18:38:17.872663 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1274> 2014-05-19 18:38:17.872674 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x71cb340 con 0x3622b00 -1273> 2014-05-19 18:38:17.872695 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1272> 2014-05-19 18:38:17.872706 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x8388000 con 0x3622dc0 -1271> 2014-05-19 18:38:17.872721 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1270> 2014-05-19 18:38:17.872727 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x73c2700 con 0x5e7ef20 -1269> 2014-05-19 18:38:17.872748 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1268> 2014-05-19 18:38:17.872759 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x6008e00 con 0x5e7f1e0 -1267> 2014-05-19 18:38:17.872790 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1266> 2014-05-19 18:38:17.872805 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x8648380 con 0x5d7bb80 -1265> 2014-05-19 18:38:17.872833 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1264> 2014-05-19 18:38:17.872851 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x5667a40 con 0x30594a0 -1263> 2014-05-19 18:38:17.872872 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1262> 2014-05-19 18:38:17.872880 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x503e000 con 0x3622f20 -1261> 2014-05-19 18:38:17.872894 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1260> 2014-05-19 18:38:17.872900 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x7a5e000 con 0x3622840 -1259> 2014-05-19 18:38:17.872933 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1258> 2014-05-19 18:38:17.872940 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x5f0b880 con 0x5e7e2c0 -1257> 2014-05-19 18:38:17.872955 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1256> 2014-05-19 18:38:17.872962 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x798e700 con 0x5e7ec60 -1255> 2014-05-19 18:38:17.872979 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1254> 2014-05-19 18:38:17.872986 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x2ee0fc0 con 0x531b760 -1253> 2014-05-19 18:38:17.873003 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1252> 2014-05-19 18:38:17.873012 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x5dfee00 con 0x53bba20 -1251> 2014-05-19 18:38:17.873027 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1250> 2014-05-19 18:38:17.873043 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x5e28fc0 con 0x51cac60 -1249> 2014-05-19 18:38:17.873058 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1248> 2014-05-19 18:38:17.873063 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x70a81c0 con 0x5bd66e0 -1247> 2014-05-19 18:38:17.873082 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1246> 2014-05-19 18:38:17.873088 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 98 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:17.872089) v2 ==== 47+0+0 (4271650009 0 0) 0x6113dc0 con 0x5bd6840 -1245> 2014-05-19 18:38:17.873102 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1244> 2014-05-19 18:38:18.121985 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:23.212173) v2 ==== 47+0+0 (2934466067 0 0) 0x3434540 con 0x3094160 -1243> 2014-05-19 18:38:18.121983 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 93 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:23.212173) v2 ==== 47+0+0 (2934466067 0 0) 0x7758540 con 0x30946e0 -1242> 2014-05-19 18:38:18.122028 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:23.212173) v2 -- ?+0 0x656cc40 con 0x3094160 -1241> 2014-05-19 18:38:18.122061 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1240> 2014-05-19 18:38:18.122078 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:23.212173) v2 -- ?+0 0x725d880 con 0x30946e0 -1239> 2014-05-19 18:38:18.122169 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1238> 2014-05-19 18:38:18.504410 7fd04f907700 5 osd.0 21 tick -1237> 2014-05-19 18:38:18.504430 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1236> 2014-05-19 18:38:18.504435 7fd04f907700 10 osd.0 21 do_waiters -- start -1235> 2014-05-19 18:38:18.504437 7fd04f907700 10 osd.0 21 do_waiters -- finish -1234> 2014-05-19 18:38:18.504467 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1233> 2014-05-19 18:38:18.675077 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:21.733259) v2 ==== 47+0+0 (2774263078 0 0) 0x6542000 con 0x4cb8dc0 -1232> 2014-05-19 18:38:18.675128 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:21.733259) v2 -- ?+0 0x7758540 con 0x4cb8dc0 -1231> 2014-05-19 18:38:18.675161 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1230> 2014-05-19 18:38:18.675152 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:21.733259) v2 ==== 47+0+0 (2774263078 0 0) 0x7735180 con 0x4cb9340 -1229> 2014-05-19 18:38:18.675221 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:21.733259) v2 -- ?+0 0x3434540 con 0x4cb9340 -1228> 2014-05-19 18:38:18.675257 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1227> 2014-05-19 18:38:18.779189 7fd04e905700 20 KeyValueStore::op_tp worker waiting -1226> 2014-05-19 18:38:18.779192 7fd043ebd700 20 OSD::op_tp worker waiting -1225> 2014-05-19 18:38:18.951263 7fd0396a8700 20 OSD::command_tp worker waiting -1224> 2014-05-19 18:38:19.345675 7fd03bead700 20 OSD::op_tp worker waiting -1223> 2014-05-19 18:38:19.345699 7fd0436bc700 20 OSD::op_tp worker waiting -1222> 2014-05-19 18:38:19.345731 7fd041eb9700 20 OSD::op_tp worker waiting -1221> 2014-05-19 18:38:19.345751 7fd03e6b2700 20 OSD::op_tp worker waiting -1220> 2014-05-19 18:38:19.345768 7fd03f6b4700 20 OSD::op_tp worker waiting -1219> 2014-05-19 18:38:19.345781 7fd03d6b0700 20 OSD::op_tp worker waiting -1218> 2014-05-19 18:38:19.345814 7fd03c6ae700 20 OSD::op_tp worker waiting -1217> 2014-05-19 18:38:19.360183 7fd0446be700 20 OSD::op_tp worker waiting -1216> 2014-05-19 18:38:19.360289 7fd03feb5700 20 OSD::op_tp worker waiting -1215> 2014-05-19 18:38:19.367416 7fd040eb7700 20 OSD::op_tp worker waiting -1214> 2014-05-19 18:38:19.372816 7fd0406b6700 20 OSD::op_tp worker waiting -1213> 2014-05-19 18:38:19.394362 7fd039ea9700 20 OSD::disk_tp worker waiting -1212> 2014-05-19 18:38:19.397988 7fd03aeab700 20 OSD::op_tp worker waiting -1211> 2014-05-19 18:38:19.398026 7fd03eeb3700 20 OSD::op_tp worker waiting -1210> 2014-05-19 18:38:19.415776 7fd042ebb700 20 OSD::op_tp worker waiting -1209> 2014-05-19 18:38:19.415802 7fd03deb1700 20 OSD::op_tp worker waiting -1208> 2014-05-19 18:38:19.424951 7fd04e104700 20 KeyValueStore::op_tp worker waiting -1207> 2014-05-19 18:38:19.424956 7fd03b6ac700 20 OSD::op_tp worker waiting -1206> 2014-05-19 18:38:19.424982 7fd0426ba700 20 OSD::op_tp worker waiting -1205> 2014-05-19 18:38:19.425002 7fd03ceaf700 20 OSD::op_tp worker waiting -1204> 2014-05-19 18:38:19.425023 7fd0416b8700 20 OSD::op_tp worker waiting -1203> 2014-05-19 18:38:19.504520 7fd04f907700 5 osd.0 21 tick -1202> 2014-05-19 18:38:19.504542 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1201> 2014-05-19 18:38:19.504546 7fd04f907700 10 osd.0 21 do_waiters -- start -1200> 2014-05-19 18:38:19.504549 7fd04f907700 10 osd.0 21 do_waiters -- finish -1199> 2014-05-19 18:38:19.504589 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1198> 2014-05-19 18:38:19.636121 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 103 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:22.694292) v2 ==== 47+0+0 (914434666 0 0) 0x64ae8c0 con 0x618fe40 -1197> 2014-05-19 18:38:19.636144 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 103 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:22.694292) v2 ==== 47+0+0 (914434666 0 0) 0x6542fc0 con 0x61ae420 -1196> 2014-05-19 18:38:19.636169 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:22.694292) v2 -- ?+0 0x7735180 con 0x618fe40 -1195> 2014-05-19 18:38:19.636202 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1194> 2014-05-19 18:38:19.636227 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:22.694292) v2 -- ?+0 0x6542000 con 0x61ae420 -1193> 2014-05-19 18:38:19.636277 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1192> 2014-05-19 18:38:19.787787 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:19.787608) v2 ==== 47+0+0 (3316497361 0 0) 0x76addc0 con 0x4cb8160 -1191> 2014-05-19 18:38:19.787841 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:19.787608) v2 -- ?+0 0x6542fc0 con 0x4cb8160 -1190> 2014-05-19 18:38:19.787866 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1189> 2014-05-19 18:38:19.787786 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:19.787608) v2 ==== 47+0+0 (3316497361 0 0) 0x60e2a80 con 0x4cb82c0 -1188> 2014-05-19 18:38:19.787953 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:19.787608) v2 -- ?+0 0x64ae8c0 con 0x4cb82c0 -1187> 2014-05-19 18:38:19.788044 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1186> 2014-05-19 18:38:19.908493 7fd0456c0700 10 monclient: tick -1185> 2014-05-19 18:38:19.908508 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:37:49.908506) -1184> 2014-05-19 18:38:19.908525 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:38:19.908525; renew after: 2014-05-19 18:38:19.906664) -- yes -1183> 2014-05-19 18:38:19.908536 7fd0456c0700 10 monclient: renew_subs -1182> 2014-05-19 18:38:19.908540 7fd0456c0700 10 monclient: _send_mon_message to mon.a at 172.16.96.201:6789/0 -1181> 2014-05-19 18:38:19.908548 7fd0456c0700 1 -- 172.16.96.201:6800/3074 --> 172.16.96.201:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0}) v2 -- ?+0 0x7570c40 con 0x2e3edc0 -1180> 2014-05-19 18:38:19.908930 7fd04aecb700 1 -- 172.16.96.201:6800/3074 <== mon.0 172.16.96.201:6789/0 46 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4198368508 0 0) 0x78908c0 con 0x2e3edc0 -1179> 2014-05-19 18:38:19.908958 7fd04aecb700 10 monclient: handle_subscribe_ack sent 2014-05-19 18:38:19.908537 renew after 2014-05-19 18:40:49.908537 -1178> 2014-05-19 18:38:20.504626 7fd04f907700 5 osd.0 21 tick -1177> 2014-05-19 18:38:20.504646 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1176> 2014-05-19 18:38:20.504651 7fd04f907700 10 osd.0 21 do_waiters -- start -1175> 2014-05-19 18:38:20.504653 7fd04f907700 10 osd.0 21 do_waiters -- finish -1174> 2014-05-19 18:38:20.504683 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1173> 2014-05-19 18:38:20.605265 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:23.663450) v2 ==== 47+0+0 (1357863992 0 0) 0x2ee01c0 con 0x61ae160 -1172> 2014-05-19 18:38:20.605298 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:23.663450) v2 ==== 47+0+0 (1357863992 0 0) 0x66f9dc0 con 0x61b0420 -1171> 2014-05-19 18:38:20.605311 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:23.663450) v2 -- ?+0 0x76addc0 con 0x61ae160 -1170> 2014-05-19 18:38:20.605362 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1169> 2014-05-19 18:38:20.605389 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:23.663450) v2 -- ?+0 0x60e2a80 con 0x61b0420 -1168> 2014-05-19 18:38:20.605430 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1167> 2014-05-19 18:38:20.772676 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -1166> 2014-05-19 18:38:20.772695 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -1165> 2014-05-19 18:38:20.772702 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x66f8700 con 0x3622580 -1164> 2014-05-19 18:38:20.772728 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x8649a40 con 0x3622840 -1163> 2014-05-19 18:38:20.772766 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x7746540 con 0x3622b00 -1162> 2014-05-19 18:38:20.772782 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x59bddc0 con 0x3622dc0 -1161> 2014-05-19 18:38:20.772797 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x682f340 con 0x30594a0 -1160> 2014-05-19 18:38:20.772811 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x51a6a80 con 0x3622f20 -1159> 2014-05-19 18:38:20.772825 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x8388540 con 0x5e7ef20 -1158> 2014-05-19 18:38:20.772848 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x7a5e1c0 con 0x5e7f1e0 -1157> 2014-05-19 18:38:20.772874 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x72e48c0 con 0x5e7e2c0 -1156> 2014-05-19 18:38:20.772905 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x6711180 con 0x5d7bb80 -1155> 2014-05-19 18:38:20.772936 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x6113500 con 0x5e7ec60 -1154> 2014-05-19 18:38:20.772963 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x6c4c000 con 0x531b760 -1153> 2014-05-19 18:38:20.772988 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x7733a40 con 0x53bba20 -1152> 2014-05-19 18:38:20.773014 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x79801c0 con 0x51cac60 -1151> 2014-05-19 18:38:20.773038 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x70a8e00 con 0x5bd66e0 -1150> 2014-05-19 18:38:20.773037 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x7aa4540 con 0x3622b00 -1149> 2014-05-19 18:38:20.773066 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:20.772701) v2 -- ?+0 0x5e28c40 con 0x5bd6840 -1148> 2014-05-19 18:38:20.773108 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1147> 2014-05-19 18:38:20.773172 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x772ce00 con 0x3622840 -1146> 2014-05-19 18:38:20.773225 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1145> 2014-05-19 18:38:20.773274 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x692dc00 con 0x3622dc0 -1144> 2014-05-19 18:38:20.773316 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1143> 2014-05-19 18:38:20.773333 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x73c2e00 con 0x5e7ef20 -1142> 2014-05-19 18:38:20.773367 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1141> 2014-05-19 18:38:20.773379 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x6008540 con 0x5e7f1e0 -1140> 2014-05-19 18:38:20.773397 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1139> 2014-05-19 18:38:20.773490 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x5f0ba40 con 0x5e7e2c0 -1138> 2014-05-19 18:38:20.773540 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1137> 2014-05-19 18:38:20.773553 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x653efc0 con 0x5d7bb80 -1136> 2014-05-19 18:38:20.773573 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -1135> 2014-05-19 18:38:20.773579 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x5f308c0 con 0x5e7ec60 -1134> 2014-05-19 18:38:20.773593 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1133> 2014-05-19 18:38:20.773598 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x55c28c0 con 0x531b760 -1132> 2014-05-19 18:38:20.773635 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1131> 2014-05-19 18:38:20.773642 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x5dfe8c0 con 0x53bba20 -1130> 2014-05-19 18:38:20.773662 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1129> 2014-05-19 18:38:20.773672 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x68aa1c0 con 0x51cac60 -1128> 2014-05-19 18:38:20.773689 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1127> 2014-05-19 18:38:20.773695 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x70a88c0 con 0x5bd66e0 -1126> 2014-05-19 18:38:20.773713 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1125> 2014-05-19 18:38:20.773723 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 99 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x5e6d6c0 con 0x5bd6840 -1124> 2014-05-19 18:38:20.773741 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1123> 2014-05-19 18:38:20.773749 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x5666a80 con 0x30594a0 -1122> 2014-05-19 18:38:20.773764 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1121> 2014-05-19 18:38:20.773771 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x6b70380 con 0x3622f20 -1120> 2014-05-19 18:38:20.773785 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1119> 2014-05-19 18:38:20.773792 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:20.772701) v2 ==== 47+0+0 (2973358908 0 0) 0x6fe7a40 con 0x3622580 -1118> 2014-05-19 18:38:20.773809 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1117> 2014-05-19 18:38:20.779285 7fd04e905700 20 KeyValueStore::op_tp worker waiting -1116> 2014-05-19 18:38:20.779287 7fd043ebd700 20 OSD::op_tp worker waiting -1115> 2014-05-19 18:38:20.951343 7fd0396a8700 20 OSD::command_tp worker waiting -1114> 2014-05-19 18:38:21.345763 7fd03bead700 20 OSD::op_tp worker waiting -1113> 2014-05-19 18:38:21.345790 7fd0436bc700 20 OSD::op_tp worker waiting -1112> 2014-05-19 18:38:21.345811 7fd03e6b2700 20 OSD::op_tp worker waiting -1111> 2014-05-19 18:38:21.345825 7fd03d6b0700 20 OSD::op_tp worker waiting -1110> 2014-05-19 18:38:21.345845 7fd041eb9700 20 OSD::op_tp worker waiting -1109> 2014-05-19 18:38:21.345875 7fd03c6ae700 20 OSD::op_tp worker waiting -1108> 2014-05-19 18:38:21.345895 7fd03f6b4700 20 OSD::op_tp worker waiting -1107> 2014-05-19 18:38:21.360272 7fd0446be700 20 OSD::op_tp worker waiting -1106> 2014-05-19 18:38:21.360361 7fd03feb5700 20 OSD::op_tp worker waiting -1105> 2014-05-19 18:38:21.367497 7fd040eb7700 20 OSD::op_tp worker waiting -1104> 2014-05-19 18:38:21.372913 7fd0406b6700 20 OSD::op_tp worker waiting -1103> 2014-05-19 18:38:21.394442 7fd039ea9700 20 OSD::disk_tp worker waiting -1102> 2014-05-19 18:38:21.398086 7fd03aeab700 20 OSD::op_tp worker waiting -1101> 2014-05-19 18:38:21.398104 7fd03eeb3700 20 OSD::op_tp worker waiting -1100> 2014-05-19 18:38:21.415875 7fd042ebb700 20 OSD::op_tp worker waiting -1099> 2014-05-19 18:38:21.415899 7fd03deb1700 20 OSD::op_tp worker waiting -1098> 2014-05-19 18:38:21.425047 7fd04e104700 20 KeyValueStore::op_tp worker waiting -1097> 2014-05-19 18:38:21.425052 7fd03b6ac700 20 OSD::op_tp worker waiting -1096> 2014-05-19 18:38:21.425074 7fd03ceaf700 20 OSD::op_tp worker waiting -1095> 2014-05-19 18:38:21.425111 7fd0416b8700 20 OSD::op_tp worker waiting -1094> 2014-05-19 18:38:21.425125 7fd0426ba700 20 OSD::op_tp worker waiting -1093> 2014-05-19 18:38:21.483921 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 103 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:26.574257) v2 ==== 47+0+0 (2398030091 0 0) 0x6e19c00 con 0x31e1600 -1092> 2014-05-19 18:38:21.483982 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.574257) v2 -- ?+0 0x2ee01c0 con 0x31e1600 -1091> 2014-05-19 18:38:21.484063 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1090> 2014-05-19 18:38:21.483929 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 103 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:26.574257) v2 ==== 47+0+0 (2398030091 0 0) 0x5dea8c0 con 0x61b0160 -1089> 2014-05-19 18:38:21.484109 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.574257) v2 -- ?+0 0x66f9dc0 con 0x61b0160 -1088> 2014-05-19 18:38:21.484140 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -1087> 2014-05-19 18:38:21.504741 7fd04f907700 5 osd.0 21 tick -1086> 2014-05-19 18:38:21.504762 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1085> 2014-05-19 18:38:21.504767 7fd04f907700 10 osd.0 21 do_waiters -- start -1084> 2014-05-19 18:38:21.504769 7fd04f907700 10 osd.0 21 do_waiters -- finish -1083> 2014-05-19 18:38:21.504793 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1082> 2014-05-19 18:38:21.702365 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:21.702157) v2 ==== 47+0+0 (2323239468 0 0) 0x73c16c0 con 0x4cb86e0 -1081> 2014-05-19 18:38:21.702385 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:21.702157) v2 ==== 47+0+0 (2323239468 0 0) 0x60a1180 con 0x4cb8420 -1080> 2014-05-19 18:38:21.702415 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:21.702157) v2 -- ?+0 0x6e19c00 con 0x4cb86e0 -1079> 2014-05-19 18:38:21.702451 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1078> 2014-05-19 18:38:21.702476 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:21.702157) v2 -- ?+0 0x5dea8c0 con 0x4cb8420 -1077> 2014-05-19 18:38:21.702505 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -1076> 2014-05-19 18:38:21.897450 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:26.987797) v2 ==== 47+0+0 (3790372125 0 0) 0x612dc00 con 0x5b21600 -1075> 2014-05-19 18:38:21.897496 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.987797) v2 -- ?+0 0x73c16c0 con 0x5b21600 -1074> 2014-05-19 18:38:21.897507 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:26.987797) v2 ==== 47+0+0 (3790372125 0 0) 0x60a16c0 con 0x5f5b760 -1073> 2014-05-19 18:38:21.897530 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1072> 2014-05-19 18:38:21.897579 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.987797) v2 -- ?+0 0x60a1180 con 0x5f5b760 -1071> 2014-05-19 18:38:21.897622 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -1070> 2014-05-19 18:38:22.504851 7fd04f907700 5 osd.0 21 tick -1069> 2014-05-19 18:38:22.504869 7fd04f907700 7 osd.0 21 do_mon_report -1068> 2014-05-19 18:38:22.504874 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -1067> 2014-05-19 18:38:22.504881 7fd04f907700 20 osd.0 21 send_pg_stats -1066> 2014-05-19 18:38:22.504892 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1065> 2014-05-19 18:38:22.504898 7fd04f907700 10 osd.0 21 do_waiters -- start -1064> 2014-05-19 18:38:22.504902 7fd04f907700 10 osd.0 21 do_waiters -- finish -1063> 2014-05-19 18:38:22.504930 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1062> 2014-05-19 18:38:22.536652 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 104 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:25.594813) v2 ==== 47+0+0 (3089028190 0 0) 0x7a36700 con 0x618fe40 -1061> 2014-05-19 18:38:22.536707 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:25.594813) v2 -- ?+0 0x60a16c0 con 0x618fe40 -1060> 2014-05-19 18:38:22.536652 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 104 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:25.594813) v2 ==== 47+0+0 (3089028190 0 0) 0x5deaa80 con 0x61ae420 -1059> 2014-05-19 18:38:22.536738 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1058> 2014-05-19 18:38:22.536773 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:25.594813) v2 -- ?+0 0x612dc00 con 0x61ae420 -1057> 2014-05-19 18:38:22.536822 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -1056> 2014-05-19 18:38:22.688366 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:22.688190) v2 ==== 47+0+0 (1302135992 0 0) 0x6aa56c0 con 0x4cb82c0 -1055> 2014-05-19 18:38:22.688352 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:22.688190) v2 ==== 47+0+0 (1302135992 0 0) 0x76adc00 con 0x4cb8160 -1054> 2014-05-19 18:38:22.688412 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:22.688190) v2 -- ?+0 0x7a36700 con 0x4cb82c0 -1053> 2014-05-19 18:38:22.688444 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1052> 2014-05-19 18:38:22.688469 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:22.688190) v2 -- ?+0 0x5deaa80 con 0x4cb8160 -1051> 2014-05-19 18:38:22.688513 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -1050> 2014-05-19 18:38:22.775608 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:25.833814) v2 ==== 47+0+0 (2179695575 0 0) 0x71cafc0 con 0x4cb8dc0 -1049> 2014-05-19 18:38:22.775662 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:25.833814) v2 -- ?+0 0x76adc00 con 0x4cb8dc0 -1048> 2014-05-19 18:38:22.775644 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:25.833814) v2 ==== 47+0+0 (2179695575 0 0) 0x7aa4380 con 0x4cb9340 -1047> 2014-05-19 18:38:22.775702 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1046> 2014-05-19 18:38:22.775755 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:25.833814) v2 -- ?+0 0x6aa56c0 con 0x4cb9340 -1045> 2014-05-19 18:38:22.775809 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -1044> 2014-05-19 18:38:22.779380 7fd04e905700 20 KeyValueStore::op_tp worker waiting -1043> 2014-05-19 18:38:22.779383 7fd043ebd700 20 OSD::op_tp worker waiting -1042> 2014-05-19 18:38:22.822234 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:27.912603) v2 ==== 47+0+0 (431827136 0 0) 0x74f6fc0 con 0x3094160 -1041> 2014-05-19 18:38:22.822227 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 94 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:27.912603) v2 ==== 47+0+0 (431827136 0 0) 0x7758700 con 0x30946e0 -1040> 2014-05-19 18:38:22.822286 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:27.912603) v2 -- ?+0 0x7aa4380 con 0x3094160 -1039> 2014-05-19 18:38:22.822326 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1038> 2014-05-19 18:38:22.822353 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:27.912603) v2 -- ?+0 0x71cafc0 con 0x30946e0 -1037> 2014-05-19 18:38:22.822386 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -1036> 2014-05-19 18:38:22.951436 7fd0396a8700 20 OSD::command_tp worker waiting -1035> 2014-05-19 18:38:23.345864 7fd03bead700 20 OSD::op_tp worker waiting -1034> 2014-05-19 18:38:23.345890 7fd03e6b2700 20 OSD::op_tp worker waiting -1033> 2014-05-19 18:38:23.345908 7fd0436bc700 20 OSD::op_tp worker waiting -1032> 2014-05-19 18:38:23.345922 7fd041eb9700 20 OSD::op_tp worker waiting -1031> 2014-05-19 18:38:23.345938 7fd03c6ae700 20 OSD::op_tp worker waiting -1030> 2014-05-19 18:38:23.345950 7fd03d6b0700 20 OSD::op_tp worker waiting -1029> 2014-05-19 18:38:23.345970 7fd03f6b4700 20 OSD::op_tp worker waiting -1028> 2014-05-19 18:38:23.360357 7fd0446be700 20 OSD::op_tp worker waiting -1027> 2014-05-19 18:38:23.360424 7fd03feb5700 20 OSD::op_tp worker waiting -1026> 2014-05-19 18:38:23.367589 7fd040eb7700 20 OSD::op_tp worker waiting -1025> 2014-05-19 18:38:23.373010 7fd0406b6700 20 OSD::op_tp worker waiting -1024> 2014-05-19 18:38:23.394526 7fd039ea9700 20 OSD::disk_tp worker waiting -1023> 2014-05-19 18:38:23.398187 7fd03aeab700 20 OSD::op_tp worker waiting -1022> 2014-05-19 18:38:23.398213 7fd03eeb3700 20 OSD::op_tp worker waiting -1021> 2014-05-19 18:38:23.415975 7fd042ebb700 20 OSD::op_tp worker waiting -1020> 2014-05-19 18:38:23.416001 7fd03deb1700 20 OSD::op_tp worker waiting -1019> 2014-05-19 18:38:23.425143 7fd04e104700 20 KeyValueStore::op_tp worker waiting -1018> 2014-05-19 18:38:23.425147 7fd03b6ac700 20 OSD::op_tp worker waiting -1017> 2014-05-19 18:38:23.425200 7fd0416b8700 20 OSD::op_tp worker waiting -1016> 2014-05-19 18:38:23.425223 7fd0426ba700 20 OSD::op_tp worker waiting -1015> 2014-05-19 18:38:23.425269 7fd03ceaf700 20 OSD::op_tp worker waiting -1014> 2014-05-19 18:38:23.504987 7fd04f907700 5 osd.0 21 tick -1013> 2014-05-19 18:38:23.505008 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -1012> 2014-05-19 18:38:23.505013 7fd04f907700 10 osd.0 21 do_waiters -- start -1011> 2014-05-19 18:38:23.505015 7fd04f907700 10 osd.0 21 do_waiters -- finish -1010> 2014-05-19 18:38:23.505056 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1009> 2014-05-19 18:38:24.505092 7fd04f907700 5 osd.0 21 tick -1008> 2014-05-19 18:38:24.505141 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -1007> 2014-05-19 18:38:24.505154 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -1006> 2014-05-19 18:38:24.505149 7fd03a6aa700 20 OSD::recovery_tp worker waiting -1005> 2014-05-19 18:38:24.505157 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 347.229 < min (86400 seconds) -1004> 2014-05-19 18:38:24.505176 7fd04f907700 20 osd.0 21 sched_scrub done -1003> 2014-05-19 18:38:24.505179 7fd04f907700 10 osd.0 21 do_waiters -- start -1002> 2014-05-19 18:38:24.505180 7fd04f907700 10 osd.0 21 do_waiters -- finish -1001> 2014-05-19 18:38:24.602930 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:24.602734) v2 ==== 47+0+0 (2786964352 0 0) 0x649e000 con 0x4cb86e0 -1000> 2014-05-19 18:38:24.602963 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:24.602734) v2 ==== 47+0+0 (2786964352 0 0) 0x60a0700 con 0x4cb8420 -999> 2014-05-19 18:38:24.602978 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:24.602734) v2 -- ?+0 0x7758700 con 0x4cb86e0 -998> 2014-05-19 18:38:24.603030 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -997> 2014-05-19 18:38:24.603056 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:24.602734) v2 -- ?+0 0x74f6fc0 con 0x4cb8420 -996> 2014-05-19 18:38:24.603101 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -995> 2014-05-19 18:38:24.705845 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:27.764141) v2 ==== 47+0+0 (2838955310 0 0) 0x70ea380 con 0x61ae160 -994> 2014-05-19 18:38:24.705898 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:27.764141) v2 -- ?+0 0x649e000 con 0x61ae160 -993> 2014-05-19 18:38:24.705903 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:27.764141) v2 ==== 47+0+0 (2838955310 0 0) 0x66f8e00 con 0x61b0420 -992> 2014-05-19 18:38:24.705934 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -991> 2014-05-19 18:38:24.705981 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:27.764141) v2 -- ?+0 0x60a0700 con 0x61b0420 -990> 2014-05-19 18:38:24.706058 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -989> 2014-05-19 18:38:24.779471 7fd04e905700 20 KeyValueStore::op_tp worker waiting -988> 2014-05-19 18:38:24.779474 7fd043ebd700 20 OSD::op_tp worker waiting -987> 2014-05-19 18:38:24.951519 7fd0396a8700 20 OSD::command_tp worker waiting -986> 2014-05-19 18:38:25.345944 7fd03e6b2700 20 OSD::op_tp worker waiting -985> 2014-05-19 18:38:25.345966 7fd03bead700 20 OSD::op_tp worker waiting -984> 2014-05-19 18:38:25.345993 7fd0436bc700 20 OSD::op_tp worker waiting -983> 2014-05-19 18:38:25.346012 7fd041eb9700 20 OSD::op_tp worker waiting -982> 2014-05-19 18:38:25.346027 7fd03d6b0700 20 OSD::op_tp worker waiting -981> 2014-05-19 18:38:25.346076 7fd03c6ae700 20 OSD::op_tp worker waiting -980> 2014-05-19 18:38:25.346131 7fd03f6b4700 20 OSD::op_tp worker waiting -979> 2014-05-19 18:38:25.360435 7fd0446be700 20 OSD::op_tp worker waiting -978> 2014-05-19 18:38:25.360489 7fd03feb5700 20 OSD::op_tp worker waiting -977> 2014-05-19 18:38:25.367684 7fd040eb7700 20 OSD::op_tp worker waiting -976> 2014-05-19 18:38:25.373107 7fd0406b6700 20 OSD::op_tp worker waiting -975> 2014-05-19 18:38:25.394618 7fd039ea9700 20 OSD::disk_tp worker waiting -974> 2014-05-19 18:38:25.398287 7fd03aeab700 20 OSD::op_tp worker waiting -973> 2014-05-19 18:38:25.398311 7fd03eeb3700 20 OSD::op_tp worker waiting -972> 2014-05-19 18:38:25.416075 7fd042ebb700 20 OSD::op_tp worker waiting -971> 2014-05-19 18:38:25.416099 7fd03deb1700 20 OSD::op_tp worker waiting -970> 2014-05-19 18:38:25.425250 7fd04e104700 20 KeyValueStore::op_tp worker waiting -969> 2014-05-19 18:38:25.425252 7fd03b6ac700 20 OSD::op_tp worker waiting -968> 2014-05-19 18:38:25.425281 7fd0416b8700 20 OSD::op_tp worker waiting -967> 2014-05-19 18:38:25.425307 7fd0426ba700 20 OSD::op_tp worker waiting -966> 2014-05-19 18:38:25.425350 7fd03ceaf700 20 OSD::op_tp worker waiting -965> 2014-05-19 18:38:25.505265 7fd04f907700 5 osd.0 21 tick -964> 2014-05-19 18:38:25.505285 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -963> 2014-05-19 18:38:25.505290 7fd04f907700 10 osd.0 21 do_waiters -- start -962> 2014-05-19 18:38:25.505293 7fd04f907700 10 osd.0 21 do_waiters -- finish -961> 2014-05-19 18:38:25.505322 7fd03a6aa700 20 OSD::recovery_tp worker waiting -960> 2014-05-19 18:38:25.584455 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 104 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:30.674918) v2 ==== 47+0+0 (914530484 0 0) 0x7aa4fc0 con 0x61b0160 -959> 2014-05-19 18:38:25.584504 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:30.674918) v2 -- ?+0 0x66f8e00 con 0x61b0160 -958> 2014-05-19 18:38:25.584612 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -957> 2014-05-19 18:38:25.584421 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 104 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:30.674918) v2 ==== 47+0+0 (914530484 0 0) 0x55f4380 con 0x31e1600 -956> 2014-05-19 18:38:25.584682 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:30.674918) v2 -- ?+0 0x70ea380 con 0x31e1600 -955> 2014-05-19 18:38:25.584713 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -954> 2014-05-19 18:38:25.676037 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 103 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:28.734309) v2 ==== 47+0+0 (3832813298 0 0) 0x3438a80 con 0x4cb8dc0 -953> 2014-05-19 18:38:25.676088 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.734309) v2 -- ?+0 0x55f4380 con 0x4cb8dc0 -952> 2014-05-19 18:38:25.676099 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 103 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:28.734309) v2 ==== 47+0+0 (3832813298 0 0) 0x7890700 con 0x4cb9340 -951> 2014-05-19 18:38:25.676126 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -950> 2014-05-19 18:38:25.676151 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.734309) v2 -- ?+0 0x7aa4fc0 con 0x4cb9340 -949> 2014-05-19 18:38:25.676187 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -948> 2014-05-19 18:38:25.997791 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:31.088298) v2 ==== 47+0+0 (2586757115 0 0) 0x799bdc0 con 0x5f5b760 -947> 2014-05-19 18:38:25.997799 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:31.088298) v2 ==== 47+0+0 (2586757115 0 0) 0x612c8c0 con 0x5b21600 -946> 2014-05-19 18:38:25.997849 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:31.088298) v2 -- ?+0 0x7890700 con 0x5f5b760 -945> 2014-05-19 18:38:25.997884 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -944> 2014-05-19 18:38:25.997955 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:31.088298) v2 -- ?+0 0x3438a80 con 0x5b21600 -943> 2014-05-19 18:38:25.997995 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -942> 2014-05-19 18:38:26.073207 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -941> 2014-05-19 18:38:26.073225 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -940> 2014-05-19 18:38:26.073237 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x66f8000 con 0x3622580 -939> 2014-05-19 18:38:26.073269 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x798f880 con 0x3622840 -938> 2014-05-19 18:38:26.073294 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x7747880 con 0x3622b00 -937> 2014-05-19 18:38:26.073323 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x59bcfc0 con 0x3622dc0 -936> 2014-05-19 18:38:26.073352 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x8648380 con 0x30594a0 -935> 2014-05-19 18:38:26.073380 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x6008e00 con 0x3622f20 -934> 2014-05-19 18:38:26.073407 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x73c2700 con 0x5e7ef20 -933> 2014-05-19 18:38:26.073446 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x8388000 con 0x5e7f1e0 -932> 2014-05-19 18:38:26.073473 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x71cb340 con 0x5e7e2c0 -931> 2014-05-19 18:38:26.073498 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x6710e00 con 0x5d7bb80 -930> 2014-05-19 18:38:26.073526 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x5667340 con 0x5e7ec60 -929> 2014-05-19 18:38:26.073552 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x6113180 con 0x531b760 -928> 2014-05-19 18:38:26.073577 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x70a9dc0 con 0x53bba20 -927> 2014-05-19 18:38:26.073601 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x5e288c0 con 0x51cac60 -926> 2014-05-19 18:38:26.073628 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x6c4d6c0 con 0x5bd66e0 -925> 2014-05-19 18:38:26.073653 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:26.073235) v2 -- ?+0 0x2ee1500 con 0x5bd6840 -924> 2014-05-19 18:38:26.073737 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x772d180 con 0x3622840 -923> 2014-05-19 18:38:26.073810 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -922> 2014-05-19 18:38:26.073826 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x669ee00 con 0x3622b00 -921> 2014-05-19 18:38:26.073887 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -920> 2014-05-19 18:38:26.073951 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x692c000 con 0x3622dc0 -919> 2014-05-19 18:38:26.073978 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -918> 2014-05-19 18:38:26.073985 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x73c3a40 con 0x5e7ef20 -917> 2014-05-19 18:38:26.074001 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -916> 2014-05-19 18:38:26.074006 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x60081c0 con 0x5e7f1e0 -915> 2014-05-19 18:38:26.074020 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -914> 2014-05-19 18:38:26.074026 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x5f0a1c0 con 0x5e7e2c0 -913> 2014-05-19 18:38:26.074040 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -912> 2014-05-19 18:38:26.074050 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x60f61c0 con 0x5d7bb80 -911> 2014-05-19 18:38:26.074065 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -910> 2014-05-19 18:38:26.074070 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x55c2380 con 0x531b760 -909> 2014-05-19 18:38:26.074084 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -908> 2014-05-19 18:38:26.074089 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x5dff340 con 0x53bba20 -907> 2014-05-19 18:38:26.074103 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -906> 2014-05-19 18:38:26.074109 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x55c9a40 con 0x5bd66e0 -905> 2014-05-19 18:38:26.074143 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -904> 2014-05-19 18:38:26.074153 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 100 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x5e6c540 con 0x5bd6840 -903> 2014-05-19 18:38:26.074171 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -902> 2014-05-19 18:38:26.074177 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x5666540 con 0x30594a0 -901> 2014-05-19 18:38:26.074191 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -900> 2014-05-19 18:38:26.074197 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x6b71dc0 con 0x3622f20 -899> 2014-05-19 18:38:26.074211 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -898> 2014-05-19 18:38:26.074216 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x6fe7880 con 0x3622580 -897> 2014-05-19 18:38:26.074231 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -896> 2014-05-19 18:38:26.074236 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x5f31880 con 0x5e7ec60 -895> 2014-05-19 18:38:26.074251 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -894> 2014-05-19 18:38:26.074259 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:26.073235) v2 ==== 47+0+0 (1188928061 0 0) 0x68ab500 con 0x51cac60 -893> 2014-05-19 18:38:26.074273 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -892> 2014-05-19 18:38:26.505374 7fd04f907700 5 osd.0 21 tick -891> 2014-05-19 18:38:26.505395 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -890> 2014-05-19 18:38:26.505400 7fd04f907700 10 osd.0 21 do_waiters -- start -889> 2014-05-19 18:38:26.505402 7fd04f907700 10 osd.0 21 do_waiters -- finish -888> 2014-05-19 18:38:26.505443 7fd03a6aa700 20 OSD::recovery_tp worker waiting -887> 2014-05-19 18:38:26.779559 7fd04e905700 20 KeyValueStore::op_tp worker waiting -886> 2014-05-19 18:38:26.779564 7fd043ebd700 20 OSD::op_tp worker waiting -885> 2014-05-19 18:38:26.922530 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:32.013078) v2 ==== 47+0+0 (4237695987 0 0) 0x7759c00 con 0x30946e0 -884> 2014-05-19 18:38:26.922545 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 95 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:32.013078) v2 ==== 47+0+0 (4237695987 0 0) 0x6b936c0 con 0x3094160 -883> 2014-05-19 18:38:26.922582 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:32.013078) v2 -- ?+0 0x612c8c0 con 0x30946e0 -882> 2014-05-19 18:38:26.922623 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -881> 2014-05-19 18:38:26.922649 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:32.013078) v2 -- ?+0 0x799bdc0 con 0x3094160 -880> 2014-05-19 18:38:26.922682 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -879> 2014-05-19 18:38:26.951614 7fd0396a8700 20 OSD::command_tp worker waiting -878> 2014-05-19 18:38:27.346007 7fd03e6b2700 20 OSD::op_tp worker waiting -877> 2014-05-19 18:38:27.346031 7fd03bead700 20 OSD::op_tp worker waiting -876> 2014-05-19 18:38:27.346061 7fd041eb9700 20 OSD::op_tp worker waiting -875> 2014-05-19 18:38:27.346089 7fd0436bc700 20 OSD::op_tp worker waiting -874> 2014-05-19 18:38:27.346112 7fd03d6b0700 20 OSD::op_tp worker waiting -873> 2014-05-19 18:38:27.346137 7fd03c6ae700 20 OSD::op_tp worker waiting -872> 2014-05-19 18:38:27.346203 7fd03f6b4700 20 OSD::op_tp worker waiting -871> 2014-05-19 18:38:27.360525 7fd0446be700 20 OSD::op_tp worker waiting -870> 2014-05-19 18:38:27.360557 7fd03feb5700 20 OSD::op_tp worker waiting -869> 2014-05-19 18:38:27.367781 7fd040eb7700 20 OSD::op_tp worker waiting -868> 2014-05-19 18:38:27.373202 7fd0406b6700 20 OSD::op_tp worker waiting -867> 2014-05-19 18:38:27.394714 7fd039ea9700 20 OSD::disk_tp worker waiting -866> 2014-05-19 18:38:27.398388 7fd03aeab700 20 OSD::op_tp worker waiting -865> 2014-05-19 18:38:27.398414 7fd03eeb3700 20 OSD::op_tp worker waiting -864> 2014-05-19 18:38:27.416175 7fd042ebb700 20 OSD::op_tp worker waiting -863> 2014-05-19 18:38:27.416202 7fd03deb1700 20 OSD::op_tp worker waiting -862> 2014-05-19 18:38:27.425356 7fd04e104700 20 KeyValueStore::op_tp worker waiting -861> 2014-05-19 18:38:27.425361 7fd03b6ac700 20 OSD::op_tp worker waiting -860> 2014-05-19 18:38:27.425390 7fd0416b8700 20 OSD::op_tp worker waiting -859> 2014-05-19 18:38:27.425410 7fd0426ba700 20 OSD::op_tp worker waiting -858> 2014-05-19 18:38:27.425428 7fd03ceaf700 20 OSD::op_tp worker waiting -857> 2014-05-19 18:38:27.505487 7fd04f907700 5 osd.0 21 tick -856> 2014-05-19 18:38:27.505504 7fd04f907700 7 osd.0 21 do_mon_report -855> 2014-05-19 18:38:27.505508 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -854> 2014-05-19 18:38:27.505511 7fd04f907700 20 osd.0 21 send_pg_stats -853> 2014-05-19 18:38:27.505519 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -852> 2014-05-19 18:38:27.505523 7fd04f907700 10 osd.0 21 do_waiters -- start -851> 2014-05-19 18:38:27.505525 7fd04f907700 10 osd.0 21 do_waiters -- finish -850> 2014-05-19 18:38:27.505564 7fd03a6aa700 20 OSD::recovery_tp worker waiting -849> 2014-05-19 18:38:27.606506 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:30.664761) v2 ==== 47+0+0 (2180702102 0 0) 0x66f9c00 con 0x61b0420 -848> 2014-05-19 18:38:27.606532 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:30.664761) v2 ==== 47+0+0 (2180702102 0 0) 0x788b500 con 0x61ae160 -847> 2014-05-19 18:38:27.606556 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:30.664761) v2 -- ?+0 0x6b936c0 con 0x61b0420 -846> 2014-05-19 18:38:27.606590 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -845> 2014-05-19 18:38:27.606632 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:30.664761) v2 -- ?+0 0x7759c00 con 0x61ae160 -844> 2014-05-19 18:38:27.606666 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -843> 2014-05-19 18:38:27.837040 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 105 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:30.895295) v2 ==== 47+0+0 (3208767878 0 0) 0x6efefc0 con 0x61ae420 -842> 2014-05-19 18:38:27.837040 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 105 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:30.895295) v2 ==== 47+0+0 (3208767878 0 0) 0x77e8000 con 0x618fe40 -841> 2014-05-19 18:38:27.837093 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:30.895295) v2 -- ?+0 0x788b500 con 0x61ae420 -840> 2014-05-19 18:38:27.837128 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -839> 2014-05-19 18:38:27.837150 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:30.895295) v2 -- ?+0 0x66f9c00 con 0x618fe40 -838> 2014-05-19 18:38:27.837216 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -837> 2014-05-19 18:38:27.989010 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:27.988837) v2 ==== 47+0+0 (668755463 0 0) 0x6aa5c00 con 0x4cb82c0 -836> 2014-05-19 18:38:27.989027 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:27.988837) v2 ==== 47+0+0 (668755463 0 0) 0x76acfc0 con 0x4cb8160 -835> 2014-05-19 18:38:27.989062 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:27.988837) v2 -- ?+0 0x77e8000 con 0x4cb82c0 -834> 2014-05-19 18:38:27.989100 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -833> 2014-05-19 18:38:27.989131 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:27.988837) v2 -- ?+0 0x6efefc0 con 0x4cb8160 -832> 2014-05-19 18:38:27.989165 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -831> 2014-05-19 18:38:28.373798 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -830> 2014-05-19 18:38:28.373816 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -829> 2014-05-19 18:38:28.373824 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x798e540 con 0x3622580 -828> 2014-05-19 18:38:28.373849 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x8649880 con 0x3622840 -827> 2014-05-19 18:38:28.373886 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x5f0b180 con 0x3622b00 -826> 2014-05-19 18:38:28.373900 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x6009180 con 0x3622dc0 -825> 2014-05-19 18:38:28.373914 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x73c21c0 con 0x30594a0 -824> 2014-05-19 18:38:28.373941 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x682fc00 con 0x3622f20 -823> 2014-05-19 18:38:28.374034 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x7a5ea80 con 0x5e7ef20 -822> 2014-05-19 18:38:28.374062 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x8389880 con 0x5e7f1e0 -821> 2014-05-19 18:38:28.374086 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x72e4700 con 0x5e7e2c0 -820> 2014-05-19 18:38:28.374110 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x6711500 con 0x5d7bb80 -819> 2014-05-19 18:38:28.374136 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x6112e00 con 0x5e7ec60 -818> 2014-05-19 18:38:28.374161 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x70a9340 con 0x531b760 -817> 2014-05-19 18:38:28.374186 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x5e28e00 con 0x53bba20 -816> 2014-05-19 18:38:28.374189 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x612ce00 con 0x3622b00 -815> 2014-05-19 18:38:28.374213 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x6c4c540 con 0x51cac60 -814> 2014-05-19 18:38:28.374288 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x2ee16c0 con 0x5bd66e0 -813> 2014-05-19 18:38:28.374312 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:28.373822) v2 -- ?+0 0x798e380 con 0x5bd6840 -812> 2014-05-19 18:38:28.374378 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -811> 2014-05-19 18:38:28.374395 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x6fe61c0 con 0x3622580 -810> 2014-05-19 18:38:28.374418 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -809> 2014-05-19 18:38:28.374430 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x772cc40 con 0x3622840 -808> 2014-05-19 18:38:28.374461 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -807> 2014-05-19 18:38:28.374475 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x692d180 con 0x3622dc0 -806> 2014-05-19 18:38:28.374505 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -805> 2014-05-19 18:38:28.374514 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x5f0aa80 con 0x5e7e2c0 -804> 2014-05-19 18:38:28.374530 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -803> 2014-05-19 18:38:28.374539 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x5666c40 con 0x30594a0 -802> 2014-05-19 18:38:28.374553 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -801> 2014-05-19 18:38:28.374559 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x6009a40 con 0x5e7f1e0 -800> 2014-05-19 18:38:28.374607 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -799> 2014-05-19 18:38:28.374624 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x60f6700 con 0x5d7bb80 -798> 2014-05-19 18:38:28.374657 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -797> 2014-05-19 18:38:28.374666 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x5f31c00 con 0x5e7ec60 -796> 2014-05-19 18:38:28.374691 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -795> 2014-05-19 18:38:28.374699 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x662bc00 con 0x531b760 -794> 2014-05-19 18:38:28.374716 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -793> 2014-05-19 18:38:28.374727 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x5dff180 con 0x53bba20 -792> 2014-05-19 18:38:28.374745 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -791> 2014-05-19 18:38:28.374753 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x68aae00 con 0x51cac60 -790> 2014-05-19 18:38:28.374768 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -789> 2014-05-19 18:38:28.374779 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x55c8700 con 0x5bd66e0 -788> 2014-05-19 18:38:28.374796 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -787> 2014-05-19 18:38:28.374803 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x6b71340 con 0x3622f20 -786> 2014-05-19 18:38:28.374818 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -785> 2014-05-19 18:38:28.374826 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x3511a40 con 0x5e7ef20 -784> 2014-05-19 18:38:28.374840 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -783> 2014-05-19 18:38:28.374846 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 101 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:28.373822) v2 ==== 47+0+0 (2382658370 0 0) 0x5e6ca80 con 0x5bd6840 -782> 2014-05-19 18:38:28.374862 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -781> 2014-05-19 18:38:28.484848 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 105 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:33.575449) v2 ==== 47+0+0 (81793989 0 0) 0x55f4fc0 con 0x31e1600 -780> 2014-05-19 18:38:28.484912 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.575449) v2 -- ?+0 0x76acfc0 con 0x31e1600 -779> 2014-05-19 18:38:28.484950 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -778> 2014-05-19 18:38:28.484853 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 105 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:33.575449) v2 ==== 47+0+0 (81793989 0 0) 0x64d2a80 con 0x61b0160 -777> 2014-05-19 18:38:28.485006 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.575449) v2 -- ?+0 0x6aa5c00 con 0x61b0160 -776> 2014-05-19 18:38:28.485036 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -775> 2014-05-19 18:38:28.505614 7fd04f907700 5 osd.0 21 tick -774> 2014-05-19 18:38:28.505662 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -773> 2014-05-19 18:38:28.505666 7fd03a6aa700 20 OSD::recovery_tp worker waiting -772> 2014-05-19 18:38:28.505676 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -771> 2014-05-19 18:38:28.505679 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 351.23 < min (86400 seconds) -770> 2014-05-19 18:38:28.505697 7fd04f907700 20 osd.0 21 sched_scrub done -769> 2014-05-19 18:38:28.505701 7fd04f907700 10 osd.0 21 do_waiters -- start -768> 2014-05-19 18:38:28.505703 7fd04f907700 10 osd.0 21 do_waiters -- finish -767> 2014-05-19 18:38:28.779644 7fd04e905700 20 KeyValueStore::op_tp worker waiting -766> 2014-05-19 18:38:28.779646 7fd043ebd700 20 OSD::op_tp worker waiting -765> 2014-05-19 18:38:28.951698 7fd0396a8700 20 OSD::command_tp worker waiting -764> 2014-05-19 18:38:29.346049 7fd03e6b2700 20 OSD::op_tp worker waiting -763> 2014-05-19 18:38:29.346073 7fd03bead700 20 OSD::op_tp worker waiting -762> 2014-05-19 18:38:29.346134 7fd041eb9700 20 OSD::op_tp worker waiting -761> 2014-05-19 18:38:29.346157 7fd03d6b0700 20 OSD::op_tp worker waiting -760> 2014-05-19 18:38:29.346178 7fd0436bc700 20 OSD::op_tp worker waiting -759> 2014-05-19 18:38:29.346203 7fd03c6ae700 20 OSD::op_tp worker waiting -758> 2014-05-19 18:38:29.346267 7fd03f6b4700 20 OSD::op_tp worker waiting -757> 2014-05-19 18:38:29.360603 7fd0446be700 20 OSD::op_tp worker waiting -756> 2014-05-19 18:38:29.360667 7fd03feb5700 20 OSD::op_tp worker waiting -755> 2014-05-19 18:38:29.367877 7fd040eb7700 20 OSD::op_tp worker waiting -754> 2014-05-19 18:38:29.373298 7fd0406b6700 20 OSD::op_tp worker waiting -753> 2014-05-19 18:38:29.394810 7fd039ea9700 20 OSD::disk_tp worker waiting -752> 2014-05-19 18:38:29.398488 7fd03aeab700 20 OSD::op_tp worker waiting -751> 2014-05-19 18:38:29.398513 7fd03eeb3700 20 OSD::op_tp worker waiting -750> 2014-05-19 18:38:29.416273 7fd042ebb700 20 OSD::op_tp worker waiting -749> 2014-05-19 18:38:29.416295 7fd03deb1700 20 OSD::op_tp worker waiting -748> 2014-05-19 18:38:29.425452 7fd04e104700 20 KeyValueStore::op_tp worker waiting -747> 2014-05-19 18:38:29.425455 7fd03b6ac700 20 OSD::op_tp worker waiting -746> 2014-05-19 18:38:29.425493 7fd0416b8700 20 OSD::op_tp worker waiting -745> 2014-05-19 18:38:29.425515 7fd0426ba700 20 OSD::op_tp worker waiting -744> 2014-05-19 18:38:29.425562 7fd03ceaf700 20 OSD::op_tp worker waiting -743> 2014-05-19 18:38:29.505788 7fd04f907700 5 osd.0 21 tick -742> 2014-05-19 18:38:29.505809 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -741> 2014-05-19 18:38:29.505813 7fd04f907700 10 osd.0 21 do_waiters -- start -740> 2014-05-19 18:38:29.505816 7fd04f907700 10 osd.0 21 do_waiters -- finish -739> 2014-05-19 18:38:29.505839 7fd03a6aa700 20 OSD::recovery_tp worker waiting -738> 2014-05-19 18:38:29.903630 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:29.903397) v2 ==== 47+0+0 (323590632 0 0) 0x79716c0 con 0x4cb8420 -737> 2014-05-19 18:38:29.903676 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:29.903397) v2 -- ?+0 0x64d2a80 con 0x4cb8420 -736> 2014-05-19 18:38:29.903721 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -735> 2014-05-19 18:38:29.903595 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:29.903397) v2 ==== 47+0+0 (323590632 0 0) 0x5bea540 con 0x4cb86e0 -734> 2014-05-19 18:38:29.903750 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:29.903397) v2 -- ?+0 0x55f4fc0 con 0x4cb86e0 -733> 2014-05-19 18:38:29.903845 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -732> 2014-05-19 18:38:29.908640 7fd0456c0700 10 monclient: tick -731> 2014-05-19 18:38:29.908652 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:37:59.908651) -730> 2014-05-19 18:38:29.908669 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:38:29.908669; renew after: 2014-05-19 18:40:49.908537) -- no -729> 2014-05-19 18:38:30.098102 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:35.188797) v2 ==== 47+0+0 (291110528 0 0) 0x649ea80 con 0x5b21600 -728> 2014-05-19 18:38:30.098154 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:35.188797) v2 -- ?+0 0x5bea540 con 0x5b21600 -727> 2014-05-19 18:38:30.098104 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:35.188797) v2 ==== 47+0+0 (291110528 0 0) 0x2f4efc0 con 0x5f5b760 -726> 2014-05-19 18:38:30.098196 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -725> 2014-05-19 18:38:30.098226 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:35.188797) v2 -- ?+0 0x79716c0 con 0x5f5b760 -724> 2014-05-19 18:38:30.098252 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -723> 2014-05-19 18:38:30.137528 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 106 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:33.195797) v2 ==== 47+0+0 (2896751024 0 0) 0x66f8e00 con 0x618fe40 -722> 2014-05-19 18:38:30.137565 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 106 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:33.195797) v2 ==== 47+0+0 (2896751024 0 0) 0x36f7500 con 0x61ae420 -721> 2014-05-19 18:38:30.137574 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.195797) v2 -- ?+0 0x2f4efc0 con 0x618fe40 -720> 2014-05-19 18:38:30.137665 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -719> 2014-05-19 18:38:30.137693 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.195797) v2 -- ?+0 0x649ea80 con 0x61ae420 -718> 2014-05-19 18:38:30.137735 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -717> 2014-05-19 18:38:30.289620 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 103 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:30.289450) v2 ==== 47+0+0 (2594922489 0 0) 0x79e6fc0 con 0x4cb8160 -716> 2014-05-19 18:38:30.289660 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 103 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:30.289450) v2 ==== 47+0+0 (2594922489 0 0) 0x656c380 con 0x4cb82c0 -715> 2014-05-19 18:38:30.289677 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:30.289450) v2 -- ?+0 0x36f7500 con 0x4cb8160 -714> 2014-05-19 18:38:30.289730 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -713> 2014-05-19 18:38:30.289789 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:30.289450) v2 -- ?+0 0x66f8e00 con 0x4cb82c0 -712> 2014-05-19 18:38:30.289838 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -711> 2014-05-19 18:38:30.505902 7fd04f907700 5 osd.0 21 tick -710> 2014-05-19 18:38:30.505923 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -709> 2014-05-19 18:38:30.505927 7fd04f907700 10 osd.0 21 do_waiters -- start -708> 2014-05-19 18:38:30.505930 7fd04f907700 10 osd.0 21 do_waiters -- finish -707> 2014-05-19 18:38:30.505960 7fd03a6aa700 20 OSD::recovery_tp worker waiting -706> 2014-05-19 18:38:30.779743 7fd04e905700 20 KeyValueStore::op_tp worker waiting -705> 2014-05-19 18:38:30.779746 7fd043ebd700 20 OSD::op_tp worker waiting -704> 2014-05-19 18:38:30.951789 7fd0396a8700 20 OSD::command_tp worker waiting -703> 2014-05-19 18:38:30.976529 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 104 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:34.034826) v2 ==== 47+0+0 (240508927 0 0) 0x3438fc0 con 0x4cb8dc0 -702> 2014-05-19 18:38:30.976543 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 104 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:34.034826) v2 ==== 47+0+0 (240508927 0 0) 0x75b1c00 con 0x4cb9340 -701> 2014-05-19 18:38:30.976593 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.034826) v2 -- ?+0 0x79e6fc0 con 0x4cb8dc0 -700> 2014-05-19 18:38:30.976619 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -699> 2014-05-19 18:38:30.976647 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.034826) v2 -- ?+0 0x656c380 con 0x4cb9340 -698> 2014-05-19 18:38:30.976683 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -697> 2014-05-19 18:38:31.022935 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:36.113622) v2 ==== 47+0+0 (2620931601 0 0) 0x6b92a80 con 0x3094160 -696> 2014-05-19 18:38:31.022937 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 96 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:36.113622) v2 ==== 47+0+0 (2620931601 0 0) 0x7759dc0 con 0x30946e0 -695> 2014-05-19 18:38:31.022976 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:36.113622) v2 -- ?+0 0x75b1c00 con 0x3094160 -694> 2014-05-19 18:38:31.023009 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -693> 2014-05-19 18:38:31.023039 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:36.113622) v2 -- ?+0 0x3438fc0 con 0x30946e0 -692> 2014-05-19 18:38:31.023095 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -691> 2014-05-19 18:38:31.346124 7fd03e6b2700 20 OSD::op_tp worker waiting -690> 2014-05-19 18:38:31.346142 7fd03bead700 20 OSD::op_tp worker waiting -689> 2014-05-19 18:38:31.346239 7fd041eb9700 20 OSD::op_tp worker waiting -688> 2014-05-19 18:38:31.346261 7fd03d6b0700 20 OSD::op_tp worker waiting -687> 2014-05-19 18:38:31.346275 7fd0436bc700 20 OSD::op_tp worker waiting -686> 2014-05-19 18:38:31.346292 7fd03c6ae700 20 OSD::op_tp worker waiting -685> 2014-05-19 18:38:31.346330 7fd03f6b4700 20 OSD::op_tp worker waiting -684> 2014-05-19 18:38:31.360699 7fd0446be700 20 OSD::op_tp worker waiting -683> 2014-05-19 18:38:31.360771 7fd03feb5700 20 OSD::op_tp worker waiting -682> 2014-05-19 18:38:31.367973 7fd040eb7700 20 OSD::op_tp worker waiting -681> 2014-05-19 18:38:31.373397 7fd0406b6700 20 OSD::op_tp worker waiting -680> 2014-05-19 18:38:31.394906 7fd039ea9700 20 OSD::disk_tp worker waiting -679> 2014-05-19 18:38:31.398591 7fd03aeab700 20 OSD::op_tp worker waiting -678> 2014-05-19 18:38:31.398651 7fd03eeb3700 20 OSD::op_tp worker waiting -677> 2014-05-19 18:38:31.416373 7fd042ebb700 20 OSD::op_tp worker waiting -676> 2014-05-19 18:38:31.416399 7fd03deb1700 20 OSD::op_tp worker waiting -675> 2014-05-19 18:38:31.425557 7fd04e104700 20 KeyValueStore::op_tp worker waiting -674> 2014-05-19 18:38:31.425561 7fd03b6ac700 20 OSD::op_tp worker waiting -673> 2014-05-19 18:38:31.425576 7fd0416b8700 20 OSD::op_tp worker waiting -672> 2014-05-19 18:38:31.425599 7fd0426ba700 20 OSD::op_tp worker waiting -671> 2014-05-19 18:38:31.425630 7fd03ceaf700 20 OSD::op_tp worker waiting -670> 2014-05-19 18:38:31.505997 7fd04f907700 5 osd.0 21 tick -669> 2014-05-19 18:38:31.506019 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -668> 2014-05-19 18:38:31.506023 7fd04f907700 10 osd.0 21 do_waiters -- start -667> 2014-05-19 18:38:31.506026 7fd04f907700 10 osd.0 21 do_waiters -- finish -666> 2014-05-19 18:38:31.506067 7fd03a6aa700 20 OSD::recovery_tp worker waiting -665> 2014-05-19 18:38:32.204084 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:32.203910) v2 ==== 47+0+0 (3835508858 0 0) 0x65421c0 con 0x4cb86e0 -664> 2014-05-19 18:38:32.204095 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:32.203910) v2 ==== 47+0+0 (3835508858 0 0) 0x5519c00 con 0x4cb8420 -663> 2014-05-19 18:38:32.204137 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:32.203910) v2 -- ?+0 0x7759dc0 con 0x4cb86e0 -662> 2014-05-19 18:38:32.204171 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -661> 2014-05-19 18:38:32.204195 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:32.203910) v2 -- ?+0 0x6b92a80 con 0x4cb8420 -660> 2014-05-19 18:38:32.204230 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -659> 2014-05-19 18:38:32.506096 7fd04f907700 5 osd.0 21 tick -658> 2014-05-19 18:38:32.506113 7fd04f907700 7 osd.0 21 do_mon_report -657> 2014-05-19 18:38:32.506117 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -656> 2014-05-19 18:38:32.506121 7fd04f907700 20 osd.0 21 send_pg_stats -655> 2014-05-19 18:38:32.506156 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -654> 2014-05-19 18:38:32.506169 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -653> 2014-05-19 18:38:32.506174 7fd03a6aa700 20 OSD::recovery_tp worker waiting -652> 2014-05-19 18:38:32.506172 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 355.23 < min (86400 seconds) -651> 2014-05-19 18:38:32.506191 7fd04f907700 20 osd.0 21 sched_scrub done -650> 2014-05-19 18:38:32.506194 7fd04f907700 10 osd.0 21 do_waiters -- start -649> 2014-05-19 18:38:32.506196 7fd04f907700 10 osd.0 21 do_waiters -- finish -648> 2014-05-19 18:38:32.779823 7fd04e905700 20 KeyValueStore::op_tp worker waiting -647> 2014-05-19 18:38:32.779845 7fd043ebd700 20 OSD::op_tp worker waiting -646> 2014-05-19 18:38:32.906959 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 103 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:35.965289) v2 ==== 47+0+0 (1964654214 0 0) 0x72dfdc0 con 0x61ae160 -645> 2014-05-19 18:38:32.907004 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 103 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:35.965289) v2 ==== 47+0+0 (1964654214 0 0) 0x5beaa80 con 0x61b0420 -644> 2014-05-19 18:38:32.907013 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:35.965289) v2 -- ?+0 0x65421c0 con 0x61ae160 -643> 2014-05-19 18:38:32.907061 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -642> 2014-05-19 18:38:32.907087 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:35.965289) v2 -- ?+0 0x5519c00 con 0x61b0420 -641> 2014-05-19 18:38:32.907142 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -640> 2014-05-19 18:38:32.951883 7fd0396a8700 20 OSD::command_tp worker waiting -639> 2014-05-19 18:38:32.998595 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:38.089329) v2 ==== 47+0+0 (908685775 0 0) 0x7970e00 con 0x5f5b760 -638> 2014-05-19 18:38:32.998608 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:38.089329) v2 ==== 47+0+0 (908685775 0 0) 0x649fa40 con 0x5b21600 -637> 2014-05-19 18:38:32.998649 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:38.089329) v2 -- ?+0 0x5beaa80 con 0x5f5b760 -636> 2014-05-19 18:38:32.998686 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -635> 2014-05-19 18:38:32.998709 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:38.089329) v2 -- ?+0 0x72dfdc0 con 0x5b21600 -634> 2014-05-19 18:38:32.998750 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -633> 2014-05-19 18:38:33.277056 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 105 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:36.335387) v2 ==== 47+0+0 (1845641113 0 0) 0x5beba40 con 0x4cb8dc0 -632> 2014-05-19 18:38:33.277073 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 105 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:36.335387) v2 ==== 47+0+0 (1845641113 0 0) 0x784d880 con 0x4cb9340 -631> 2014-05-19 18:38:33.277105 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:36.335387) v2 -- ?+0 0x649fa40 con 0x4cb8dc0 -630> 2014-05-19 18:38:33.277143 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -629> 2014-05-19 18:38:33.277165 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:36.335387) v2 -- ?+0 0x7970e00 con 0x4cb9340 -628> 2014-05-19 18:38:33.277206 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -627> 2014-05-19 18:38:33.346221 7fd03e6b2700 20 OSD::op_tp worker waiting -626> 2014-05-19 18:38:33.346244 7fd03bead700 20 OSD::op_tp worker waiting -625> 2014-05-19 18:38:33.346317 7fd041eb9700 20 OSD::op_tp worker waiting -624> 2014-05-19 18:38:33.346349 7fd03d6b0700 20 OSD::op_tp worker waiting -623> 2014-05-19 18:38:33.346370 7fd03f6b4700 20 OSD::op_tp worker waiting -622> 2014-05-19 18:38:33.346387 7fd0436bc700 20 OSD::op_tp worker waiting -621> 2014-05-19 18:38:33.346447 7fd03c6ae700 20 OSD::op_tp worker waiting -620> 2014-05-19 18:38:33.360801 7fd0446be700 20 OSD::op_tp worker waiting -619> 2014-05-19 18:38:33.360853 7fd03feb5700 20 OSD::op_tp worker waiting -618> 2014-05-19 18:38:33.368056 7fd040eb7700 20 OSD::op_tp worker waiting -617> 2014-05-19 18:38:33.373495 7fd0406b6700 20 OSD::op_tp worker waiting -616> 2014-05-19 18:38:33.395001 7fd039ea9700 20 OSD::disk_tp worker waiting -615> 2014-05-19 18:38:33.398690 7fd03aeab700 20 OSD::op_tp worker waiting -614> 2014-05-19 18:38:33.398729 7fd03eeb3700 20 OSD::op_tp worker waiting -613> 2014-05-19 18:38:33.416471 7fd042ebb700 20 OSD::op_tp worker waiting -612> 2014-05-19 18:38:33.416495 7fd03deb1700 20 OSD::op_tp worker waiting -611> 2014-05-19 18:38:33.425658 7fd04e104700 20 KeyValueStore::op_tp worker waiting -610> 2014-05-19 18:38:33.425659 7fd03b6ac700 20 OSD::op_tp worker waiting -609> 2014-05-19 18:38:33.425695 7fd0416b8700 20 OSD::op_tp worker waiting -608> 2014-05-19 18:38:33.425716 7fd0426ba700 20 OSD::op_tp worker waiting -607> 2014-05-19 18:38:33.425765 7fd03ceaf700 20 OSD::op_tp worker waiting -606> 2014-05-19 18:38:33.506273 7fd04f907700 5 osd.0 21 tick -605> 2014-05-19 18:38:33.506295 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -604> 2014-05-19 18:38:33.506299 7fd04f907700 10 osd.0 21 do_waiters -- start -603> 2014-05-19 18:38:33.506302 7fd04f907700 10 osd.0 21 do_waiters -- finish -602> 2014-05-19 18:38:33.506343 7fd03a6aa700 20 OSD::recovery_tp worker waiting -601> 2014-05-19 18:38:33.674456 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -600> 2014-05-19 18:38:33.674474 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -599> 2014-05-19 18:38:33.674482 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x8648e00 con 0x3622580 -598> 2014-05-19 18:38:33.674508 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x5f0a380 con 0x3622840 -597> 2014-05-19 18:38:33.674546 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x60088c0 con 0x3622b00 -596> 2014-05-19 18:38:33.674560 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x51a68c0 con 0x3622dc0 -595> 2014-05-19 18:38:33.674593 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x5f0a1c0 con 0x30594a0 -594> 2014-05-19 18:38:33.674625 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x60081c0 con 0x3622f20 -593> 2014-05-19 18:38:33.674653 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x73c3a40 con 0x5e7ef20 -592> 2014-05-19 18:38:33.674682 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x692c000 con 0x5e7f1e0 -591> 2014-05-19 18:38:33.674706 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x669ee00 con 0x5e7e2c0 -590> 2014-05-19 18:38:33.674729 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x772d180 con 0x5d7bb80 -589> 2014-05-19 18:38:33.674754 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x6fe7a40 con 0x5e7ec60 -588> 2014-05-19 18:38:33.674779 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x6b70380 con 0x531b760 -587> 2014-05-19 18:38:33.674803 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x5666a80 con 0x53bba20 -586> 2014-05-19 18:38:33.674844 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x5e6d6c0 con 0x51cac60 -585> 2014-05-19 18:38:33.674872 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x70a88c0 con 0x5bd66e0 -584> 2014-05-19 18:38:33.674898 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:33.674480) v2 -- ?+0 0x68aa1c0 con 0x5bd6840 -583> 2014-05-19 18:38:33.674903 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x7735340 con 0x3622b00 -582> 2014-05-19 18:38:33.675015 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -581> 2014-05-19 18:38:33.675037 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x772d340 con 0x3622840 -580> 2014-05-19 18:38:33.675076 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -579> 2014-05-19 18:38:33.675092 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x692d340 con 0x3622dc0 -578> 2014-05-19 18:38:33.675111 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -577> 2014-05-19 18:38:33.675120 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x3511500 con 0x5e7ef20 -576> 2014-05-19 18:38:33.675174 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -575> 2014-05-19 18:38:33.675186 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x6008000 con 0x5e7f1e0 -574> 2014-05-19 18:38:33.675207 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -573> 2014-05-19 18:38:33.675214 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x662ba40 con 0x531b760 -572> 2014-05-19 18:38:33.675228 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -571> 2014-05-19 18:38:33.675239 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x6c4a000 con 0x30594a0 -570> 2014-05-19 18:38:33.675253 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -569> 2014-05-19 18:38:33.675259 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x6b71500 con 0x3622f20 -568> 2014-05-19 18:38:33.675273 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -567> 2014-05-19 18:38:33.675279 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x7930700 con 0x3622580 -566> 2014-05-19 18:38:33.675293 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -565> 2014-05-19 18:38:33.675298 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x7733340 con 0x5e7e2c0 -564> 2014-05-19 18:38:33.675313 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -563> 2014-05-19 18:38:33.675320 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x60f68c0 con 0x5d7bb80 -562> 2014-05-19 18:38:33.675334 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -561> 2014-05-19 18:38:33.675343 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x5f30e00 con 0x5e7ec60 -560> 2014-05-19 18:38:33.675357 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -559> 2014-05-19 18:38:33.675363 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x5dff880 con 0x53bba20 -558> 2014-05-19 18:38:33.675380 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -557> 2014-05-19 18:38:33.675387 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x64a6540 con 0x51cac60 -556> 2014-05-19 18:38:33.675401 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -555> 2014-05-19 18:38:33.675409 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x55c9500 con 0x5bd66e0 -554> 2014-05-19 18:38:33.675422 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -553> 2014-05-19 18:38:33.675430 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 102 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:33.674480) v2 ==== 47+0+0 (727916957 0 0) 0x5e6d340 con 0x5bd6840 -552> 2014-05-19 18:38:33.675445 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -551> 2014-05-19 18:38:33.785046 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 106 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:38.875921) v2 ==== 47+0+0 (2707360125 0 0) 0x6504700 con 0x61b0160 -550> 2014-05-19 18:38:33.785058 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 106 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:38.875921) v2 ==== 47+0+0 (2707360125 0 0) 0x5f7b340 con 0x31e1600 -549> 2014-05-19 18:38:33.785101 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:38.875921) v2 -- ?+0 0x784d880 con 0x61b0160 -548> 2014-05-19 18:38:33.785140 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -547> 2014-05-19 18:38:33.785166 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:38.875921) v2 -- ?+0 0x5beba40 con 0x31e1600 -546> 2014-05-19 18:38:33.785209 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -545> 2014-05-19 18:38:33.923288 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:39.014102) v2 ==== 47+0+0 (1399415585 0 0) 0x6b92700 con 0x3094160 -544> 2014-05-19 18:38:33.923329 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:39.014102) v2 -- ?+0 0x6504700 con 0x3094160 -543> 2014-05-19 18:38:33.923281 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 97 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:39.014102) v2 ==== 47+0+0 (1399415585 0 0) 0x7758e00 con 0x30946e0 -542> 2014-05-19 18:38:33.923364 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -541> 2014-05-19 18:38:33.923384 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:39.014102) v2 -- ?+0 0x5f7b340 con 0x30946e0 -540> 2014-05-19 18:38:33.923434 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -539> 2014-05-19 18:38:34.506368 7fd04f907700 5 osd.0 21 tick -538> 2014-05-19 18:38:34.506389 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -537> 2014-05-19 18:38:34.506394 7fd04f907700 10 osd.0 21 do_waiters -- start -536> 2014-05-19 18:38:34.506396 7fd04f907700 10 osd.0 21 do_waiters -- finish -535> 2014-05-19 18:38:34.506437 7fd03a6aa700 20 OSD::recovery_tp worker waiting -534> 2014-05-19 18:38:34.775048 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -533> 2014-05-19 18:38:34.775067 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -532> 2014-05-19 18:38:34.775075 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x5dfe8c0 con 0x3622580 -531> 2014-05-19 18:38:34.775100 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x55c28c0 con 0x3622840 -530> 2014-05-19 18:38:34.775137 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x5f308c0 con 0x3622b00 -529> 2014-05-19 18:38:34.775151 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x653efc0 con 0x3622dc0 -528> 2014-05-19 18:38:34.775186 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x5f0ba40 con 0x30594a0 -527> 2014-05-19 18:38:34.775216 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x6008540 con 0x3622f20 -526> 2014-05-19 18:38:34.775252 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x73c2e00 con 0x5e7ef20 -525> 2014-05-19 18:38:34.775278 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x692dc00 con 0x5e7f1e0 -524> 2014-05-19 18:38:34.775304 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x772ce00 con 0x5e7e2c0 -523> 2014-05-19 18:38:34.775330 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x7aa4540 con 0x5d7bb80 -522> 2014-05-19 18:38:34.775356 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x6113dc0 con 0x5e7ec60 -521> 2014-05-19 18:38:34.775382 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x70a81c0 con 0x531b760 -520> 2014-05-19 18:38:34.775430 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x5e28fc0 con 0x53bba20 -519> 2014-05-19 18:38:34.775456 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x5dfee00 con 0x51cac60 -518> 2014-05-19 18:38:34.775479 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x2ee0fc0 con 0x5bd66e0 -517> 2014-05-19 18:38:34.775470 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x5f7b180 con 0x3622840 -516> 2014-05-19 18:38:34.775504 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:34.775073) v2 -- ?+0 0x798e700 con 0x5bd6840 -515> 2014-05-19 18:38:34.775573 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -514> 2014-05-19 18:38:34.775608 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x5b28540 con 0x3622580 -513> 2014-05-19 18:38:34.775648 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -512> 2014-05-19 18:38:34.775668 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x6543500 con 0x3622b00 -511> 2014-05-19 18:38:34.775703 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -510> 2014-05-19 18:38:34.775727 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x692da40 con 0x3622dc0 -509> 2014-05-19 18:38:34.775758 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -508> 2014-05-19 18:38:34.775768 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x60096c0 con 0x5e7f1e0 -507> 2014-05-19 18:38:34.775783 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -506> 2014-05-19 18:38:34.775790 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x7732540 con 0x5e7e2c0 -505> 2014-05-19 18:38:34.775807 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -504> 2014-05-19 18:38:34.775813 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x60f6e00 con 0x5d7bb80 -503> 2014-05-19 18:38:34.775827 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -502> 2014-05-19 18:38:34.775832 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x6c4a700 con 0x30594a0 -501> 2014-05-19 18:38:34.775855 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -500> 2014-05-19 18:38:34.775871 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x6b71c00 con 0x3622f20 -499> 2014-05-19 18:38:34.775899 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -498> 2014-05-19 18:38:34.775918 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x6d61500 con 0x5e7ef20 -497> 2014-05-19 18:38:34.775947 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -496> 2014-05-19 18:38:34.775966 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x662b880 con 0x531b760 -495> 2014-05-19 18:38:34.775997 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -494> 2014-05-19 18:38:34.776007 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x5dffdc0 con 0x53bba20 -493> 2014-05-19 18:38:34.776025 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -492> 2014-05-19 18:38:34.776037 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x64a7a40 con 0x51cac60 -491> 2014-05-19 18:38:34.776054 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -490> 2014-05-19 18:38:34.776062 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x5cbca80 con 0x5bd66e0 -489> 2014-05-19 18:38:34.776080 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -488> 2014-05-19 18:38:34.776087 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 103 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x5497dc0 con 0x5bd6840 -487> 2014-05-19 18:38:34.776101 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -486> 2014-05-19 18:38:34.776109 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:34.775073) v2 ==== 47+0+0 (3580840984 0 0) 0x5f30540 con 0x5e7ec60 -485> 2014-05-19 18:38:34.776123 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -484> 2014-05-19 18:38:34.779902 7fd04e905700 20 KeyValueStore::op_tp worker waiting -483> 2014-05-19 18:38:34.779921 7fd043ebd700 20 OSD::op_tp worker waiting -482> 2014-05-19 18:38:34.951964 7fd0396a8700 20 OSD::command_tp worker waiting -481> 2014-05-19 18:38:35.207531 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 104 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:38.265898) v2 ==== 47+0+0 (2197847890 0 0) 0x79628c0 con 0x61ae160 -480> 2014-05-19 18:38:35.207584 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:38.265898) v2 -- ?+0 0x7758e00 con 0x61ae160 -479> 2014-05-19 18:38:35.207586 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 104 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:38.265898) v2 ==== 47+0+0 (2197847890 0 0) 0x5bea700 con 0x61b0420 -478> 2014-05-19 18:38:35.207676 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -477> 2014-05-19 18:38:35.207739 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:38.265898) v2 -- ?+0 0x6b92700 con 0x61b0420 -476> 2014-05-19 18:38:35.207768 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -475> 2014-05-19 18:38:35.346292 7fd03e6b2700 20 OSD::op_tp worker waiting -474> 2014-05-19 18:38:35.346317 7fd03bead700 20 OSD::op_tp worker waiting -473> 2014-05-19 18:38:35.346418 7fd041eb9700 20 OSD::op_tp worker waiting -472> 2014-05-19 18:38:35.346443 7fd03d6b0700 20 OSD::op_tp worker waiting -471> 2014-05-19 18:38:35.346461 7fd03f6b4700 20 OSD::op_tp worker waiting -470> 2014-05-19 18:38:35.346511 7fd0436bc700 20 OSD::op_tp worker waiting -469> 2014-05-19 18:38:35.346541 7fd03c6ae700 20 OSD::op_tp worker waiting -468> 2014-05-19 18:38:35.360889 7fd0446be700 20 OSD::op_tp worker waiting -467> 2014-05-19 18:38:35.360917 7fd03feb5700 20 OSD::op_tp worker waiting -466> 2014-05-19 18:38:35.368138 7fd040eb7700 20 OSD::op_tp worker waiting -465> 2014-05-19 18:38:35.373592 7fd0406b6700 20 OSD::op_tp worker waiting -464> 2014-05-19 18:38:35.395083 7fd039ea9700 20 OSD::disk_tp worker waiting -463> 2014-05-19 18:38:35.398789 7fd03aeab700 20 OSD::op_tp worker waiting -462> 2014-05-19 18:38:35.398808 7fd03eeb3700 20 OSD::op_tp worker waiting -461> 2014-05-19 18:38:35.416570 7fd042ebb700 20 OSD::op_tp worker waiting -460> 2014-05-19 18:38:35.416616 7fd03deb1700 20 OSD::op_tp worker waiting -459> 2014-05-19 18:38:35.425766 7fd04e104700 20 KeyValueStore::op_tp worker waiting -458> 2014-05-19 18:38:35.425770 7fd03b6ac700 20 OSD::op_tp worker waiting -457> 2014-05-19 18:38:35.425792 7fd0416b8700 20 OSD::op_tp worker waiting -456> 2014-05-19 18:38:35.425813 7fd0426ba700 20 OSD::op_tp worker waiting -455> 2014-05-19 18:38:35.425842 7fd03ceaf700 20 OSD::op_tp worker waiting -454> 2014-05-19 18:38:35.437982 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 107 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:38.496346) v2 ==== 47+0+0 (1932892077 0 0) 0x784d6c0 con 0x61ae420 -453> 2014-05-19 18:38:35.438033 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:38.496346) v2 -- ?+0 0x79628c0 con 0x61ae420 -452> 2014-05-19 18:38:35.438028 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 107 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:38.496346) v2 ==== 47+0+0 (1932892077 0 0) 0x66f9dc0 con 0x618fe40 -451> 2014-05-19 18:38:35.438066 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -450> 2014-05-19 18:38:35.438143 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:38.496346) v2 -- ?+0 0x5bea700 con 0x618fe40 -449> 2014-05-19 18:38:35.438192 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -448> 2014-05-19 18:38:35.506475 7fd04f907700 5 osd.0 21 tick -447> 2014-05-19 18:38:35.506496 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -446> 2014-05-19 18:38:35.506500 7fd04f907700 10 osd.0 21 do_waiters -- start -445> 2014-05-19 18:38:35.506503 7fd04f907700 10 osd.0 21 do_waiters -- finish -444> 2014-05-19 18:38:35.506544 7fd03a6aa700 20 OSD::recovery_tp worker waiting -443> 2014-05-19 18:38:35.590199 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 104 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:35.590030) v2 ==== 47+0+0 (3760972576 0 0) 0x79e76c0 con 0x4cb8160 -442> 2014-05-19 18:38:35.590246 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:35.590030) v2 -- ?+0 0x784d6c0 con 0x4cb8160 -441> 2014-05-19 18:38:35.590286 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -440> 2014-05-19 18:38:35.590260 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 104 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:35.590030) v2 ==== 47+0+0 (3760972576 0 0) 0x656d880 con 0x4cb82c0 -439> 2014-05-19 18:38:35.590353 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:35.590030) v2 -- ?+0 0x66f9dc0 con 0x4cb82c0 -438> 2014-05-19 18:38:35.590439 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -437> 2014-05-19 18:38:36.085494 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 107 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:41.176400) v2 ==== 47+0+0 (256121769 0 0) 0x7930540 con 0x31e1600 -436> 2014-05-19 18:38:36.085542 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:41.176400) v2 -- ?+0 0x79e76c0 con 0x31e1600 -435> 2014-05-19 18:38:36.085578 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -434> 2014-05-19 18:38:36.085545 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 107 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:41.176400) v2 ==== 47+0+0 (256121769 0 0) 0x692c000 con 0x61b0160 -433> 2014-05-19 18:38:36.085641 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:41.176400) v2 -- ?+0 0x656d880 con 0x61b0160 -432> 2014-05-19 18:38:36.085678 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -431> 2014-05-19 18:38:36.506570 7fd04f907700 5 osd.0 21 tick -430> 2014-05-19 18:38:36.506591 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -429> 2014-05-19 18:38:36.506595 7fd04f907700 10 osd.0 21 do_waiters -- start -428> 2014-05-19 18:38:36.506598 7fd04f907700 10 osd.0 21 do_waiters -- finish -427> 2014-05-19 18:38:36.506638 7fd03a6aa700 20 OSD::recovery_tp worker waiting -426> 2014-05-19 18:38:36.538472 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 108 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:39.596844) v2 ==== 47+0+0 (2845544048 0 0) 0x784c380 con 0x61ae420 -425> 2014-05-19 18:38:36.538472 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 108 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:39.596844) v2 ==== 47+0+0 (2845544048 0 0) 0x55c88c0 con 0x618fe40 -424> 2014-05-19 18:38:36.538514 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:39.596844) v2 -- ?+0 0x7930540 con 0x61ae420 -423> 2014-05-19 18:38:36.538553 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -422> 2014-05-19 18:38:36.538594 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:39.596844) v2 -- ?+0 0x692c000 con 0x618fe40 -421> 2014-05-19 18:38:36.538667 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -420> 2014-05-19 18:38:36.690782 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 105 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:36.690609) v2 ==== 47+0+0 (569807193 0 0) 0x79e68c0 con 0x4cb8160 -419> 2014-05-19 18:38:36.690807 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 105 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:36.690609) v2 ==== 47+0+0 (569807193 0 0) 0x656ca80 con 0x4cb82c0 -418> 2014-05-19 18:38:36.690831 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:36.690609) v2 -- ?+0 0x784c380 con 0x4cb8160 -417> 2014-05-19 18:38:36.690866 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -416> 2014-05-19 18:38:36.690906 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:36.690609) v2 -- ?+0 0x55c88c0 con 0x4cb82c0 -415> 2014-05-19 18:38:36.691001 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -414> 2014-05-19 18:38:36.779998 7fd04e905700 20 KeyValueStore::op_tp worker waiting -413> 2014-05-19 18:38:36.780000 7fd043ebd700 20 OSD::op_tp worker waiting -412> 2014-05-19 18:38:36.952044 7fd0396a8700 20 OSD::command_tp worker waiting -411> 2014-05-19 18:38:37.346361 7fd03e6b2700 20 OSD::op_tp worker waiting -410> 2014-05-19 18:38:37.346387 7fd03bead700 20 OSD::op_tp worker waiting -409> 2014-05-19 18:38:37.346479 7fd041eb9700 20 OSD::op_tp worker waiting -408> 2014-05-19 18:38:37.346503 7fd03d6b0700 20 OSD::op_tp worker waiting -407> 2014-05-19 18:38:37.346528 7fd03f6b4700 20 OSD::op_tp worker waiting -406> 2014-05-19 18:38:37.346575 7fd0436bc700 20 OSD::op_tp worker waiting -405> 2014-05-19 18:38:37.346612 7fd03c6ae700 20 OSD::op_tp worker waiting -404> 2014-05-19 18:38:37.360975 7fd0446be700 20 OSD::op_tp worker waiting -403> 2014-05-19 18:38:37.361029 7fd03feb5700 20 OSD::op_tp worker waiting -402> 2014-05-19 18:38:37.368220 7fd040eb7700 20 OSD::op_tp worker waiting -401> 2014-05-19 18:38:37.373689 7fd0406b6700 20 OSD::op_tp worker waiting -400> 2014-05-19 18:38:37.395163 7fd039ea9700 20 OSD::disk_tp worker waiting -399> 2014-05-19 18:38:37.398891 7fd03aeab700 20 OSD::op_tp worker waiting -398> 2014-05-19 18:38:37.398951 7fd03eeb3700 20 OSD::op_tp worker waiting -397> 2014-05-19 18:38:37.416688 7fd042ebb700 20 OSD::op_tp worker waiting -396> 2014-05-19 18:38:37.416714 7fd03deb1700 20 OSD::op_tp worker waiting -395> 2014-05-19 18:38:37.425864 7fd04e104700 20 KeyValueStore::op_tp worker waiting -394> 2014-05-19 18:38:37.425868 7fd03b6ac700 20 OSD::op_tp worker waiting -393> 2014-05-19 18:38:37.425906 7fd0416b8700 20 OSD::op_tp worker waiting -392> 2014-05-19 18:38:37.425927 7fd0426ba700 20 OSD::op_tp worker waiting -391> 2014-05-19 18:38:37.425976 7fd03ceaf700 20 OSD::op_tp worker waiting -390> 2014-05-19 18:38:37.504666 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:37.504452) v2 ==== 47+0+0 (3985484967 0 0) 0x6543c00 con 0x4cb86e0 -389> 2014-05-19 18:38:37.504704 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 100 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:37.504452) v2 ==== 47+0+0 (3985484967 0 0) 0x72de380 con 0x4cb8420 -388> 2014-05-19 18:38:37.504714 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.504452) v2 -- ?+0 0x79e68c0 con 0x4cb86e0 -387> 2014-05-19 18:38:37.504829 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -386> 2014-05-19 18:38:37.504880 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.504452) v2 -- ?+0 0x656ca80 con 0x4cb8420 -385> 2014-05-19 18:38:37.504925 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -384> 2014-05-19 18:38:37.506684 7fd04f907700 5 osd.0 21 tick -383> 2014-05-19 18:38:37.506701 7fd04f907700 7 osd.0 21 do_mon_report -382> 2014-05-19 18:38:37.506704 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -381> 2014-05-19 18:38:37.506708 7fd04f907700 20 osd.0 21 send_pg_stats -380> 2014-05-19 18:38:37.506744 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -379> 2014-05-19 18:38:37.506757 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -378> 2014-05-19 18:38:37.506753 7fd03a6aa700 20 OSD::recovery_tp worker waiting -377> 2014-05-19 18:38:37.506760 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 360.231 < min (86400 seconds) -376> 2014-05-19 18:38:37.506787 7fd04f907700 20 osd.0 21 sched_scrub done -375> 2014-05-19 18:38:37.506790 7fd04f907700 10 osd.0 21 do_waiters -- start -374> 2014-05-19 18:38:37.506792 7fd04f907700 10 osd.0 21 do_waiters -- finish -373> 2014-05-19 18:38:37.675640 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -372> 2014-05-19 18:38:37.675658 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -371> 2014-05-19 18:38:37.675666 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x5f0b880 con 0x3622580 -370> 2014-05-19 18:38:37.675692 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x7a5e000 con 0x3622840 -369> 2014-05-19 18:38:37.675730 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x503e000 con 0x3622b00 -368> 2014-05-19 18:38:37.675745 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x5667a40 con 0x3622dc0 -367> 2014-05-19 18:38:37.675769 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x662ba40 con 0x30594a0 -366> 2014-05-19 18:38:37.675800 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x6008000 con 0x3622f20 -365> 2014-05-19 18:38:37.675829 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x3511500 con 0x5e7ef20 -364> 2014-05-19 18:38:37.675859 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x692d340 con 0x5e7f1e0 -363> 2014-05-19 18:38:37.675886 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x772d340 con 0x5e7e2c0 -362> 2014-05-19 18:38:37.675915 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x7735340 con 0x5d7bb80 -361> 2014-05-19 18:38:37.675942 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x5e6ca80 con 0x5e7ec60 -360> 2014-05-19 18:38:37.675970 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x3511a40 con 0x531b760 -359> 2014-05-19 18:38:37.676015 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x6b71340 con 0x53bba20 -358> 2014-05-19 18:38:37.676041 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x55c8700 con 0x51cac60 -357> 2014-05-19 18:38:37.676071 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x68aae00 con 0x5bd66e0 -356> 2014-05-19 18:38:37.676097 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:37.675665) v2 -- ?+0 0x5dff180 con 0x5bd6840 -355> 2014-05-19 18:38:37.676094 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x788b180 con 0x3622b00 -354> 2014-05-19 18:38:37.676139 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -353> 2014-05-19 18:38:37.676152 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x6d616c0 con 0x3622dc0 -352> 2014-05-19 18:38:37.676182 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -351> 2014-05-19 18:38:37.676290 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x7980380 con 0x3622580 -350> 2014-05-19 18:38:37.676340 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -349> 2014-05-19 18:38:37.676357 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x72e5dc0 con 0x3622840 -348> 2014-05-19 18:38:37.676376 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -347> 2014-05-19 18:38:37.676390 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x7971500 con 0x5e7f1e0 -346> 2014-05-19 18:38:37.676419 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -345> 2014-05-19 18:38:37.676431 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x6009500 con 0x5e7e2c0 -344> 2014-05-19 18:38:37.676452 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -343> 2014-05-19 18:38:37.676474 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x7999880 con 0x5d7bb80 -342> 2014-05-19 18:38:37.676503 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -341> 2014-05-19 18:38:37.676513 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x77e8c40 con 0x5e7ec60 -340> 2014-05-19 18:38:37.676530 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -339> 2014-05-19 18:38:37.676538 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x798fdc0 con 0x531b760 -338> 2014-05-19 18:38:37.676556 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -337> 2014-05-19 18:38:37.676563 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x73c0540 con 0x53bba20 -336> 2014-05-19 18:38:37.676592 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -335> 2014-05-19 18:38:37.676601 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x6c4d180 con 0x51cac60 -334> 2014-05-19 18:38:37.676619 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -333> 2014-05-19 18:38:37.676624 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x5cbc380 con 0x5bd66e0 -332> 2014-05-19 18:38:37.676639 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -331> 2014-05-19 18:38:37.676648 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 104 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x5497340 con 0x5bd6840 -330> 2014-05-19 18:38:37.676665 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -329> 2014-05-19 18:38:37.676673 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x6d60700 con 0x30594a0 -328> 2014-05-19 18:38:37.676688 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -327> 2014-05-19 18:38:37.676695 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x59bcfc0 con 0x3622f20 -326> 2014-05-19 18:38:37.676709 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -325> 2014-05-19 18:38:37.676717 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:37.675665) v2 ==== 47+0+0 (4040245899 0 0) 0x73c2540 con 0x5e7ef20 -324> 2014-05-19 18:38:37.676734 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -323> 2014-05-19 18:38:38.298870 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:43.389855) v2 ==== 47+0+0 (4141710540 0 0) 0x72e4fc0 con 0x5b21600 -322> 2014-05-19 18:38:38.298920 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:43.389855) v2 -- ?+0 0x6543c00 con 0x5b21600 -321> 2014-05-19 18:38:38.298925 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:43.389855) v2 ==== 47+0+0 (4141710540 0 0) 0x711e700 con 0x5f5b760 -320> 2014-05-19 18:38:38.298958 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -319> 2014-05-19 18:38:38.299018 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:43.389855) v2 -- ?+0 0x72de380 con 0x5f5b760 -318> 2014-05-19 18:38:38.299058 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -317> 2014-05-19 18:38:38.506879 7fd04f907700 5 osd.0 21 tick -316> 2014-05-19 18:38:38.506928 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -315> 2014-05-19 18:38:38.506941 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -314> 2014-05-19 18:38:38.506948 7fd03a6aa700 20 OSD::recovery_tp worker waiting -313> 2014-05-19 18:38:38.506944 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 361.231 < min (86400 seconds) -312> 2014-05-19 18:38:38.506963 7fd04f907700 20 osd.0 21 sched_scrub done -311> 2014-05-19 18:38:38.506966 7fd04f907700 10 osd.0 21 do_waiters -- start -310> 2014-05-19 18:38:38.506968 7fd04f907700 10 osd.0 21 do_waiters -- finish -309> 2014-05-19 18:38:38.577485 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 106 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:41.635879) v2 ==== 47+0+0 (42693298 0 0) 0x5c041c0 con 0x4cb9340 -308> 2014-05-19 18:38:38.577493 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 106 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:41.635879) v2 ==== 47+0+0 (42693298 0 0) 0x7aa5340 con 0x4cb8dc0 -307> 2014-05-19 18:38:38.577534 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:41.635879) v2 -- ?+0 0x711e700 con 0x4cb9340 -306> 2014-05-19 18:38:38.577571 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -305> 2014-05-19 18:38:38.577669 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:41.635879) v2 -- ?+0 0x72e4fc0 con 0x4cb8dc0 -304> 2014-05-19 18:38:38.577710 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -303> 2014-05-19 18:38:38.605186 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:38.605014) v2 ==== 47+0+0 (2169465157 0 0) 0x6543880 con 0x4cb86e0 -302> 2014-05-19 18:38:38.605218 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 101 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:38.605014) v2 ==== 47+0+0 (2169465157 0 0) 0x70cc1c0 con 0x4cb8420 -301> 2014-05-19 18:38:38.605262 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:38.605014) v2 -- ?+0 0x7aa5340 con 0x4cb86e0 -300> 2014-05-19 18:38:38.605298 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -299> 2014-05-19 18:38:38.605350 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:38.605014) v2 -- ?+0 0x5c041c0 con 0x4cb8420 -298> 2014-05-19 18:38:38.605390 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -297> 2014-05-19 18:38:38.780094 7fd04e905700 20 KeyValueStore::op_tp worker waiting -296> 2014-05-19 18:38:38.780095 7fd043ebd700 20 OSD::op_tp worker waiting -295> 2014-05-19 18:38:38.952132 7fd0396a8700 20 OSD::command_tp worker waiting -294> 2014-05-19 18:38:39.223578 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:44.314595) v2 ==== 47+0+0 (598975114 0 0) 0x77596c0 con 0x30946e0 -293> 2014-05-19 18:38:39.223602 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 98 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:44.314595) v2 ==== 47+0+0 (598975114 0 0) 0x6b928c0 con 0x3094160 -292> 2014-05-19 18:38:39.223626 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:44.314595) v2 -- ?+0 0x6543880 con 0x30946e0 -291> 2014-05-19 18:38:39.223657 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -290> 2014-05-19 18:38:39.223699 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:44.314595) v2 -- ?+0 0x70cc1c0 con 0x3094160 -289> 2014-05-19 18:38:39.223747 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -288> 2014-05-19 18:38:39.346441 7fd03e6b2700 20 OSD::op_tp worker waiting -287> 2014-05-19 18:38:39.346466 7fd03bead700 20 OSD::op_tp worker waiting -286> 2014-05-19 18:38:39.346559 7fd041eb9700 20 OSD::op_tp worker waiting -285> 2014-05-19 18:38:39.346578 7fd03d6b0700 20 OSD::op_tp worker waiting -284> 2014-05-19 18:38:39.346592 7fd03f6b4700 20 OSD::op_tp worker waiting -283> 2014-05-19 18:38:39.346638 7fd0436bc700 20 OSD::op_tp worker waiting -282> 2014-05-19 18:38:39.346676 7fd03c6ae700 20 OSD::op_tp worker waiting -281> 2014-05-19 18:38:39.361069 7fd0446be700 20 OSD::op_tp worker waiting -280> 2014-05-19 18:38:39.361097 7fd03feb5700 20 OSD::op_tp worker waiting -279> 2014-05-19 18:38:39.368300 7fd040eb7700 20 OSD::op_tp worker waiting -278> 2014-05-19 18:38:39.373787 7fd0406b6700 20 OSD::op_tp worker waiting -277> 2014-05-19 18:38:39.395245 7fd039ea9700 20 OSD::disk_tp worker waiting -276> 2014-05-19 18:38:39.398975 7fd03aeab700 20 OSD::op_tp worker waiting -275> 2014-05-19 18:38:39.399031 7fd03eeb3700 20 OSD::op_tp worker waiting -274> 2014-05-19 18:38:39.416788 7fd042ebb700 20 OSD::op_tp worker waiting -273> 2014-05-19 18:38:39.416815 7fd03deb1700 20 OSD::op_tp worker waiting -272> 2014-05-19 18:38:39.425971 7fd04e104700 20 KeyValueStore::op_tp worker waiting -271> 2014-05-19 18:38:39.425976 7fd03b6ac700 20 OSD::op_tp worker waiting -270> 2014-05-19 18:38:39.426000 7fd0416b8700 20 OSD::op_tp worker waiting -269> 2014-05-19 18:38:39.426019 7fd0426ba700 20 OSD::op_tp worker waiting -268> 2014-05-19 18:38:39.426091 7fd03ceaf700 20 OSD::op_tp worker waiting -267> 2014-05-19 18:38:39.438951 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.21 172.16.96.202:0/2458 109 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:42.497348) v2 ==== 47+0+0 (3612180455 0 0) 0x6c4da40 con 0x61ae420 -266> 2014-05-19 18:38:39.438958 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.21 172.16.96.202:0/2458 109 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:42.497348) v2 ==== 47+0+0 (3612180455 0 0) 0x799a700 con 0x618fe40 -265> 2014-05-19 18:38:39.439005 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.497348) v2 -- ?+0 0x77596c0 con 0x61ae420 -264> 2014-05-19 18:38:39.439079 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -263> 2014-05-19 18:38:39.439114 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2458 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.497348) v2 -- ?+0 0x6b928c0 con 0x618fe40 -262> 2014-05-19 18:38:39.439149 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -261> 2014-05-19 18:38:39.507044 7fd04f907700 5 osd.0 21 tick -260> 2014-05-19 18:38:39.507064 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -259> 2014-05-19 18:38:39.507069 7fd04f907700 10 osd.0 21 do_waiters -- start -258> 2014-05-19 18:38:39.507071 7fd04f907700 10 osd.0 21 do_waiters -- finish -257> 2014-05-19 18:38:39.507112 7fd03a6aa700 20 OSD::recovery_tp worker waiting -256> 2014-05-19 18:38:39.591343 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.1 172.16.96.201:0/3279 106 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:39.591174) v2 ==== 47+0+0 (1687631405 0 0) 0x656c540 con 0x4cb8160 -255> 2014-05-19 18:38:39.591390 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:39.591174) v2 -- ?+0 0x6c4da40 con 0x4cb8160 -254> 2014-05-19 18:38:39.591415 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -253> 2014-05-19 18:38:39.591404 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.1 172.16.96.201:0/3279 106 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:39.591174) v2 ==== 47+0+0 (1687631405 0 0) 0x5deb500 con 0x4cb82c0 -252> 2014-05-19 18:38:39.591460 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3279 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:39.591174) v2 -- ?+0 0x799a700 con 0x4cb82c0 -251> 2014-05-19 18:38:39.591540 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -250> 2014-05-19 18:38:39.677975 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 107 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:42.736373) v2 ==== 47+0+0 (2672163432 0 0) 0x6aa56c0 con 0x4cb8dc0 -249> 2014-05-19 18:38:39.677993 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 107 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:42.736373) v2 ==== 47+0+0 (2672163432 0 0) 0x5c04000 con 0x4cb9340 -248> 2014-05-19 18:38:39.678014 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.736373) v2 -- ?+0 0x656c540 con 0x4cb8dc0 -247> 2014-05-19 18:38:39.678053 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -246> 2014-05-19 18:38:39.678073 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.736373) v2 -- ?+0 0x5deb500 con 0x4cb9340 -245> 2014-05-19 18:38:39.678103 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -244> 2014-05-19 18:38:39.908772 7fd0456c0700 10 monclient: tick -243> 2014-05-19 18:38:39.908786 7fd0456c0700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-05-19 18:38:09.908784) -242> 2014-05-19 18:38:39.908803 7fd0456c0700 10 monclient: renew subs? (now: 2014-05-19 18:38:39.908803; renew after: 2014-05-19 18:40:49.908537) -- no -241> 2014-05-19 18:38:40.507115 7fd04f907700 5 osd.0 21 tick -240> 2014-05-19 18:38:40.507135 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -239> 2014-05-19 18:38:40.507140 7fd04f907700 10 osd.0 21 do_waiters -- start -238> 2014-05-19 18:38:40.507142 7fd04f907700 10 osd.0 21 do_waiters -- finish -237> 2014-05-19 18:38:40.507171 7fd03a6aa700 20 OSD::recovery_tp worker waiting -236> 2014-05-19 18:38:40.508177 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 105 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:43.566571) v2 ==== 47+0+0 (3343126907 0 0) 0x55c21c0 con 0x61ae160 -235> 2014-05-19 18:38:40.508218 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:43.566571) v2 -- ?+0 0x6aa56c0 con 0x61ae160 -234> 2014-05-19 18:38:40.508265 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 105 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:43.566571) v2 ==== 47+0+0 (3343126907 0 0) 0x70cd500 con 0x61b0420 -233> 2014-05-19 18:38:40.508298 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -232> 2014-05-19 18:38:40.508350 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:43.566571) v2 -- ?+0 0x5c04000 con 0x61b0420 -231> 2014-05-19 18:38:40.508392 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -230> 2014-05-19 18:38:40.599236 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:45.690339) v2 ==== 47+0+0 (1068137269 0 0) 0x2ee1dc0 con 0x5f5b760 -229> 2014-05-19 18:38:40.599267 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:45.690339) v2 ==== 47+0+0 (1068137269 0 0) 0x72e41c0 con 0x5b21600 -228> 2014-05-19 18:38:40.599320 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:45.690339) v2 -- ?+0 0x70cd500 con 0x5f5b760 -227> 2014-05-19 18:38:40.599395 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -226> 2014-05-19 18:38:40.599421 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:45.690339) v2 -- ?+0 0x55c21c0 con 0x5b21600 -225> 2014-05-19 18:38:40.599469 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -224> 2014-05-19 18:38:40.780193 7fd04e905700 20 KeyValueStore::op_tp worker waiting -223> 2014-05-19 18:38:40.780198 7fd043ebd700 20 OSD::op_tp worker waiting -222> 2014-05-19 18:38:40.952224 7fd0396a8700 20 OSD::command_tp worker waiting -221> 2014-05-19 18:38:41.346499 7fd03e6b2700 20 OSD::op_tp worker waiting -220> 2014-05-19 18:38:41.346517 7fd03bead700 20 OSD::op_tp worker waiting -219> 2014-05-19 18:38:41.346616 7fd041eb9700 20 OSD::op_tp worker waiting -218> 2014-05-19 18:38:41.346641 7fd03d6b0700 20 OSD::op_tp worker waiting -217> 2014-05-19 18:38:41.346658 7fd03f6b4700 20 OSD::op_tp worker waiting -216> 2014-05-19 18:38:41.346701 7fd0436bc700 20 OSD::op_tp worker waiting -215> 2014-05-19 18:38:41.346739 7fd03c6ae700 20 OSD::op_tp worker waiting -214> 2014-05-19 18:38:41.361149 7fd0446be700 20 OSD::op_tp worker waiting -213> 2014-05-19 18:38:41.361171 7fd03feb5700 20 OSD::op_tp worker waiting -212> 2014-05-19 18:38:41.368384 7fd040eb7700 20 OSD::op_tp worker waiting -211> 2014-05-19 18:38:41.373885 7fd0406b6700 20 OSD::op_tp worker waiting -210> 2014-05-19 18:38:41.385805 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 108 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:46.476917) v2 ==== 47+0+0 (2983874675 0 0) 0x692cc40 con 0x61b0160 -209> 2014-05-19 18:38:41.385855 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:46.476917) v2 -- ?+0 0x2ee1dc0 con 0x61b0160 -208> 2014-05-19 18:38:41.385845 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 108 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:46.476917) v2 ==== 47+0+0 (2983874675 0 0) 0x7931a40 con 0x31e1600 -207> 2014-05-19 18:38:41.385908 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -206> 2014-05-19 18:38:41.385933 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:46.476917) v2 -- ?+0 0x72e41c0 con 0x31e1600 -205> 2014-05-19 18:38:41.385974 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -204> 2014-05-19 18:38:41.395328 7fd039ea9700 20 OSD::disk_tp worker waiting -203> 2014-05-19 18:38:41.399059 7fd03aeab700 20 OSD::op_tp worker waiting -202> 2014-05-19 18:38:41.399114 7fd03eeb3700 20 OSD::op_tp worker waiting -201> 2014-05-19 18:38:41.416889 7fd042ebb700 20 OSD::op_tp worker waiting -200> 2014-05-19 18:38:41.416915 7fd03deb1700 20 OSD::op_tp worker waiting -199> 2014-05-19 18:38:41.426067 7fd04e104700 20 KeyValueStore::op_tp worker waiting -198> 2014-05-19 18:38:41.426071 7fd03b6ac700 20 OSD::op_tp worker waiting -197> 2014-05-19 18:38:41.426095 7fd0416b8700 20 OSD::op_tp worker waiting -196> 2014-05-19 18:38:41.426118 7fd0426ba700 20 OSD::op_tp worker waiting -195> 2014-05-19 18:38:41.426156 7fd03ceaf700 20 OSD::op_tp worker waiting -194> 2014-05-19 18:38:41.505811 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.2 172.16.96.201:0/3485 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:41.505624) v2 ==== 47+0+0 (3871721343 0 0) 0x71cb180 con 0x4cb86e0 -193> 2014-05-19 18:38:41.505853 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:41.505624) v2 -- ?+0 0x7931a40 con 0x4cb86e0 -192> 2014-05-19 18:38:41.505852 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.2 172.16.96.201:0/3485 102 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:41.505624) v2 ==== 47+0+0 (3871721343 0 0) 0x784ca80 con 0x4cb8420 -191> 2014-05-19 18:38:41.505932 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -190> 2014-05-19 18:38:41.505954 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.201:0/3485 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:41.505624) v2 -- ?+0 0x692cc40 con 0x4cb8420 -189> 2014-05-19 18:38:41.506057 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -188> 2014-05-19 18:38:41.507204 7fd04f907700 5 osd.0 21 tick -187> 2014-05-19 18:38:41.507264 7fd03a6aa700 20 OSD::recovery_tp worker waiting -186> 2014-05-19 18:38:41.507261 7fd04f907700 20 osd.0 21 scrub_should_schedule loadavg 0.01 < max 0.5 = yes -185> 2014-05-19 18:38:41.507279 7fd04f907700 20 osd.0 21 sched_scrub load_is_low=1 -184> 2014-05-19 18:38:41.507284 7fd04f907700 10 osd.0 21 sched_scrub 0.4 at 2014-05-19 18:32:37.275963: 364.231 < min (86400 seconds) -183> 2014-05-19 18:38:41.507324 7fd04f907700 20 osd.0 21 sched_scrub done -182> 2014-05-19 18:38:41.507330 7fd04f907700 10 osd.0 21 do_waiters -- start -181> 2014-05-19 18:38:41.507333 7fd04f907700 10 osd.0 21 do_waiters -- finish -180> 2014-05-19 18:38:41.523875 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.41 172.16.96.203:0/2619 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:46.615093) v2 ==== 47+0+0 (2996774179 0 0) 0x6b92000 con 0x3094160 -179> 2014-05-19 18:38:41.523908 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:46.615093) v2 -- ?+0 0x784ca80 con 0x3094160 -178> 2014-05-19 18:38:41.523927 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -177> 2014-05-19 18:38:41.523916 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.41 172.16.96.203:0/2619 99 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:46.615093) v2 ==== 47+0+0 (2996774179 0 0) 0x5e24a80 con 0x30946e0 -176> 2014-05-19 18:38:41.523975 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2619 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:46.615093) v2 -- ?+0 0x71cb180 con 0x30946e0 -175> 2014-05-19 18:38:41.524009 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -174> 2014-05-19 18:38:41.608691 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.22 172.16.96.202:0/2726 106 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:44.667127) v2 ==== 47+0+0 (622590474 0 0) 0x51a6540 con 0x61ae160 -173> 2014-05-19 18:38:41.608719 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.22 172.16.96.202:0/2726 106 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:44.667127) v2 ==== 47+0+0 (622590474 0 0) 0x71221c0 con 0x61b0420 -172> 2014-05-19 18:38:41.608741 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:44.667127) v2 -- ?+0 0x5e24a80 con 0x61ae160 -171> 2014-05-19 18:38:41.608783 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -170> 2014-05-19 18:38:41.608807 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2726 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:44.667127) v2 -- ?+0 0x6b92000 con 0x61b0420 -169> 2014-05-19 18:38:41.608836 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -168> 2014-05-19 18:38:42.376238 7fd038ea7700 20 osd.0 21 update_osd_stat osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -167> 2014-05-19 18:38:42.376257 7fd038ea7700 5 osd.0 21 heartbeat: osd_stat(43948 kB used, 931 GB avail, 931 GB total, peers [1,2,20,21,22,40,41,42]/[] op hist []) -166> 2014-05-19 18:38:42.376265 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6807/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x662bc00 con 0x3622580 -165> 2014-05-19 18:38:42.376290 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6808/3279 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x5f31c00 con 0x3622840 -164> 2014-05-19 18:38:42.376329 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6812/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x60f6700 con 0x3622b00 -163> 2014-05-19 18:38:42.376344 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.201:6813/3485 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x6009a40 con 0x3622dc0 -162> 2014-05-19 18:38:42.376366 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6802/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x5666c40 con 0x30594a0 -161> 2014-05-19 18:38:42.376407 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6803/2218 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x5f0aa80 con 0x3622f20 -160> 2014-05-19 18:38:42.376436 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6807/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x692d180 con 0x5e7ef20 -159> 2014-05-19 18:38:42.376462 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6808/2458 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x772cc40 con 0x5e7f1e0 -158> 2014-05-19 18:38:42.376485 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6812/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x6fe61c0 con 0x5e7e2c0 -157> 2014-05-19 18:38:42.376508 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.202:6813/2726 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x612ce00 con 0x5d7bb80 -156> 2014-05-19 18:38:42.376531 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6802/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x68ab500 con 0x5e7ec60 -155> 2014-05-19 18:38:42.376555 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6803/2353 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x5f31880 con 0x531b760 -154> 2014-05-19 18:38:42.376623 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6807/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x6fe7880 con 0x53bba20 -153> 2014-05-19 18:38:42.376645 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6808/2619 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x6b71dc0 con 0x51cac60 -152> 2014-05-19 18:38:42.376668 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6812/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x5666540 con 0x5bd66e0 -151> 2014-05-19 18:38:42.376693 7fd038ea7700 1 -- 172.16.96.201:0/3074 --> 172.16.96.203:6813/2923 -- osd_ping(ping e21 stamp 2014-05-19 18:38:42.376263) v2 -- ?+0 0x5e6c540 con 0x5bd6840 -150> 2014-05-19 18:38:42.376703 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6807/3279 107 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x79801c0 con 0x3622580 -149> 2014-05-19 18:38:42.376738 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -148> 2014-05-19 18:38:42.376755 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6812/3485 107 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x788a000 con 0x3622b00 -147> 2014-05-19 18:38:42.376871 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -146> 2014-05-19 18:38:42.376897 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.1 172.16.96.201:6808/3279 107 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x72e4540 con 0x3622840 -145> 2014-05-19 18:38:42.376918 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3058160 already has epoch 21 -144> 2014-05-19 18:38:42.376959 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.2 172.16.96.201:6813/3485 107 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x6d61180 con 0x3622dc0 -143> 2014-05-19 18:38:42.376987 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x5b78580 already has epoch 21 -142> 2014-05-19 18:38:42.376996 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6808/2458 107 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x682f340 con 0x5e7f1e0 -141> 2014-05-19 18:38:42.377015 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -140> 2014-05-19 18:38:42.377024 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6812/2726 107 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x60081c0 con 0x5e7e2c0 -139> 2014-05-19 18:38:42.377040 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -138> 2014-05-19 18:38:42.377046 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.22 172.16.96.202:6813/2726 107 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x7998c40 con 0x5d7bb80 -137> 2014-05-19 18:38:42.377095 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7e40 already has epoch 21 -136> 2014-05-19 18:38:42.377109 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6802/2353 107 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x77e8e00 con 0x5e7ec60 -135> 2014-05-19 18:38:42.377139 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -134> 2014-05-19 18:38:42.377149 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.40 172.16.96.203:6803/2353 107 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x798e1c0 con 0x531b760 -133> 2014-05-19 18:38:42.377181 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -132> 2014-05-19 18:38:42.377194 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6807/2619 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x6e188c0 con 0x53bba20 -131> 2014-05-19 18:38:42.377209 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -130> 2014-05-19 18:38:42.377220 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.41 172.16.96.203:6808/2619 106 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x8388700 con 0x51cac60 -129> 2014-05-19 18:38:42.377237 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -128> 2014-05-19 18:38:42.377244 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6812/2923 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x5cbc000 con 0x5bd66e0 -127> 2014-05-19 18:38:42.377260 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -126> 2014-05-19 18:38:42.377269 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.42 172.16.96.203:6813/2923 105 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x5496c40 con 0x5bd6840 -125> 2014-05-19 18:38:42.377286 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -124> 2014-05-19 18:38:42.377294 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6802/2218 107 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x68aae00 con 0x30594a0 -123> 2014-05-19 18:38:42.377309 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -122> 2014-05-19 18:38:42.377317 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.20 172.16.96.202:6803/2218 107 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x59bddc0 con 0x3622f20 -121> 2014-05-19 18:38:42.377331 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -120> 2014-05-19 18:38:42.377339 7fd048ec7700 1 -- 172.16.96.201:0/3074 <== osd.21 172.16.96.202:6807/2458 107 ==== osd_ping(ping_reply e21 stamp 2014-05-19 18:38:42.376263) v2 ==== 47+0+0 (1010333788 0 0) 0x73c2380 con 0x5e7ef20 -119> 2014-05-19 18:38:42.377355 7fd048ec7700 20 osd.0 21 _share_map_outgoing 0x4cb9760 already has epoch 21 -118> 2014-05-19 18:38:42.486327 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.40 172.16.96.203:0/2353 109 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:47.577512) v2 ==== 47+0+0 (1721509144 0 0) 0x772d180 con 0x31e1600 -117> 2014-05-19 18:38:42.486379 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:47.577512) v2 -- ?+0 0x51a6540 con 0x31e1600 -116> 2014-05-19 18:38:42.486410 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -115> 2014-05-19 18:38:42.486335 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.40 172.16.96.203:0/2353 109 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:47.577512) v2 ==== 47+0+0 (1721509144 0 0) 0x692d500 con 0x61b0160 -114> 2014-05-19 18:38:42.486503 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2353 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:47.577512) v2 -- ?+0 0x71221c0 con 0x61b0160 -113> 2014-05-19 18:38:42.486537 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -112> 2014-05-19 18:38:42.507410 7fd04f907700 5 osd.0 21 tick -111> 2014-05-19 18:38:42.507426 7fd04f907700 7 osd.0 21 do_mon_report -110> 2014-05-19 18:38:42.507430 7fd04f907700 10 osd.0 21 send_alive up_thru currently 20 want 20 -109> 2014-05-19 18:38:42.507434 7fd04f907700 20 osd.0 21 send_pg_stats -108> 2014-05-19 18:38:42.507441 7fd04f907700 20 osd.0 21 scrub_random_backoff lost coin flip, randomly backing off -107> 2014-05-19 18:38:42.507445 7fd04f907700 10 osd.0 21 do_waiters -- start -106> 2014-05-19 18:38:42.507447 7fd04f907700 10 osd.0 21 do_waiters -- finish -105> 2014-05-19 18:38:42.507461 7fd03a6aa700 20 OSD::recovery_tp worker waiting -104> 2014-05-19 18:38:42.578457 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.20 172.16.96.202:0/2218 108 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:45.636913) v2 ==== 47+0+0 (1636555968 0 0) 0x5c05a40 con 0x4cb9340 -103> 2014-05-19 18:38:42.578469 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.20 172.16.96.202:0/2218 108 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:45.636913) v2 ==== 47+0+0 (1636555968 0 0) 0x3434540 con 0x4cb8dc0 -102> 2014-05-19 18:38:42.578513 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:45.636913) v2 -- ?+0 0x692d500 con 0x4cb9340 -101> 2014-05-19 18:38:42.578548 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -100> 2014-05-19 18:38:42.578598 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.202:0/2218 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:45.636913) v2 -- ?+0 0x772d180 con 0x4cb8dc0 -99> 2014-05-19 18:38:42.578636 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -98> 2014-05-19 18:38:42.780287 7fd04e905700 20 KeyValueStore::op_tp worker waiting -97> 2014-05-19 18:38:42.780289 7fd043ebd700 20 OSD::op_tp worker waiting -96> 2014-05-19 18:38:42.899675 7fd0486c6700 1 -- 172.16.96.201:6803/3074 <== osd.42 172.16.96.203:0/2923 103 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:47.990862) v2 ==== 47+0+0 (329170888 0 0) 0x2ee08c0 con 0x5f5b760 -95> 2014-05-19 18:38:42.899703 7fd0476c4700 1 -- 172.16.96.201:6802/3074 <== osd.42 172.16.96.203:0/2923 103 ==== osd_ping(ping e21 stamp 2014-05-19 18:38:47.990862) v2 ==== 47+0+0 (329170888 0 0) 0x6aa5880 con 0x5b21600 -94> 2014-05-19 18:38:42.899725 7fd0486c6700 1 -- 172.16.96.201:6803/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:47.990862) v2 -- ?+0 0x5c05a40 con 0x5f5b760 -93> 2014-05-19 18:38:42.899766 7fd0486c6700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -92> 2014-05-19 18:38:42.899804 7fd0476c4700 1 -- 172.16.96.201:6802/3074 --> 172.16.96.203:0/2923 -- osd_ping(ping_reply e21 stamp 2014-05-19 18:38:47.990862) v2 -- ?+0 0x3434540 con 0x5b21600 -91> 2014-05-19 18:38:42.899847 7fd0476c4700 20 osd.0 21 _share_map_outgoing 0x31e1b80 already has epoch 21 -90> 2014-05-19 18:38:42.952317 7fd0396a8700 20 OSD::command_tp worker waiting -89> 2014-05-19 18:38:43.001828 7fd02ef84700 1 -- 172.16.96.201:6800/3074 >> :/0 pipe(0x6b9b900 sd=66 :6800 s=0 pgs=0 cs=0 l=0 c=0x61fba20).accept sd=66 172.16.96.219:52635/0 -88> 2014-05-19 18:38:43.001961 7fd02ef84700 10 osd.0 21 new session 0x2de9a40 con=0x61fba20 addr=172.16.96.219:0/1052969 -87> 2014-05-19 18:38:43.027008 7fd04aecb700 1 -- 172.16.96.201:6800/3074 <== client.4133 172.16.96.219:0/1052969 1 ==== osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 ==== 227+0+4194304 (3842731600 0 0) 0x66b8fc0 con 0x61fba20 -86> 2014-05-19 18:38:43.027060 7fd04aecb700 10 osd.0 21 do_waiters -- start -85> 2014-05-19 18:38:43.027066 7fd04aecb700 10 osd.0 21 do_waiters -- finish -84> 2014-05-19 18:38:43.027069 7fd04aecb700 20 osd.0 21 _dispatch 0x66b8fc0 osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 -83> 2014-05-19 18:38:43.027086 7fd04aecb700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.006140, event: header_read, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 -82> 2014-05-19 18:38:43.027109 7fd04aecb700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.006146, event: throttled, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 -81> 2014-05-19 18:38:43.027139 7fd04aecb700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.026948, event: all_read, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 -80> 2014-05-19 18:38:43.027159 7fd04aecb700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.027056, event: dispatched, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 -79> 2014-05-19 18:38:43.027181 7fd04aecb700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.027181, event: waiting_for_osdmap, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 -78> 2014-05-19 18:38:43.027206 7fd04aecb700 15 osd.0 21 require_same_or_newer_map 21 (i am 21) 0x66b8fc0 -77> 2014-05-19 18:38:43.027213 7fd04aecb700 20 osd.0 21 _share_map_incoming client.4133 172.16.96.219:0/1052969 21 -76> 2014-05-19 18:38:43.027242 7fd04aecb700 15 osd.0 21 enqueue_op 0x5aaf2c0 prio 63 cost 4194304 latency 0.021101 osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 -75> 2014-05-19 18:38:43.027277 7fd04aecb700 10 osd.0 21 do_waiters -- start -74> 2014-05-19 18:38:43.027282 7fd04aecb700 10 osd.0 21 do_waiters -- finish -73> 2014-05-19 18:38:43.027291 7fd03e6b2700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active) -72> 2014-05-19 18:38:43.027304 7fd03e6b2700 10 osd.0 21 dequeue_op 0x5aaf2c0 prio 63 cost 4194304 latency 0.021164 osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 pg pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] -71> 2014-05-19 18:38:43.027331 7fd03e6b2700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.027331, event: reached_pg, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 -70> 2014-05-19 18:38:43.027355 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -69> 2014-05-19 18:38:43.027373 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] handle_message: 0x5aaf2c0 -68> 2014-05-19 18:38:43.027387 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_op osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 may_write -> write-ordered flags ack+ondisk+write -67> 2014-05-19 18:38:43.027426 7fd03e6b2700 15 getattr 3.3f_head/d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 '_' -66> 2014-05-19 18:38:43.027497 7fd03e6b2700 10 getattr got.size() is 0 -65> 2014-05-19 18:38:43.027503 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: no obc for soid d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 but can_create -64> 2014-05-19 18:38:43.027522 7fd03e6b2700 15 getattr 3.3f_head/d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 'snapset' -63> 2014-05-19 18:38:43.027549 7fd03e6b2700 10 getattr got.size() is 0 -62> 2014-05-19 18:38:43.027552 7fd03e6b2700 15 getattr 3.3f_head/d310c43f/rbd_data.10236b8b4567.0000000000000004/snapdir//3 'snapset' -61> 2014-05-19 18:38:43.027576 7fd03e6b2700 10 getattr got.size() is 0 -60> 2014-05-19 18:38:43.027589 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] create_object_context 0x77ad080 d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 -59> 2014-05-19 18:38:43.027606 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] populate_obc_watchers d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 -58> 2014-05-19 18:38:43.027618 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 -57> 2014-05-19 18:38:43.027629 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: 0x77ad080 d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 rwstate(none n=0 w=0) oi: d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x2de8fc0 snapset: 0=[]:[] -56> 2014-05-19 18:38:43.027649 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] find_object_context d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 @head oi=d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) -55> 2014-05-19 18:38:43.027674 7fd03e6b2700 15 getattr 3.3f_head/d310c43f/rbd_data.10236b8b4567.0000000000000004/snapdir//3 '_' -54> 2014-05-19 18:38:43.027705 7fd03e6b2700 10 getattr got.size() is 0 -53> 2014-05-19 18:38:43.027707 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: no obc for soid d310c43f/rbd_data.10236b8b4567.0000000000000004/snapdir//3 and !can_create -52> 2014-05-19 18:38:43.027722 7fd03e6b2700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.027722, event: started, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 -51> 2014-05-19 18:38:43.027742 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] execute_ctx 0x6b0b600 -50> 2014-05-19 18:38:43.027767 7fd03e6b2700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.027767, event: started, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 -49> 2014-05-19 18:38:43.027782 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_op d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] ov 0'0 av 21'1 snapc 0=[] snapset 0=[]:[] -48> 2014-05-19 18:38:43.027803 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] -47> 2014-05-19 18:38:43.027816 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 -46> 2014-05-19 18:38:43.027837 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op write 0~4194304 -45> 2014-05-19 18:38:43.027863 7fd03e6b2700 15 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op_effects on session 0x2de9a40 -44> 2014-05-19 18:38:43.027881 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] make_writeable d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 snapset=0x2de8ff8 snapc=0=[] -43> 2014-05-19 18:38:43.027894 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] setting DIRTY flag -42> 2014-05-19 18:38:43.027904 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] make_writeable d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 done, snapset=0=[]:[]+head -41> 2014-05-19 18:38:43.027917 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] finish_ctx d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 0x6b0b600 op modify -40> 2014-05-19 18:38:43.027932 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] set mtime to 2014-05-19 12:37:55.600025 -39> 2014-05-19 18:38:43.027964 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] final snapset 0=[]:[]+head in d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 -38> 2014-05-19 18:38:43.027985 7fd03e6b2700 20 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] zeroing write result code 0 -37> 2014-05-19 18:38:43.027996 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] new_repop rep_tid 1 on osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 -36> 2014-05-19 18:38:43.028023 7fd03e6b2700 7 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] issue_repop rep_tid 1 o d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 -35> 2014-05-19 18:38:43.028048 7fd03e6b2700 5 -- op tracker -- , seq: 23062, time: 2014-05-19 18:38:43.028048, event: waiting for subops from 41, request: osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4 -34> 2014-05-19 18:38:43.028082 7fd03e6b2700 20 osd.0 21 _share_map_outgoing 0x31e1760 already has epoch 21 -33> 2014-05-19 18:38:43.028088 7fd03e6b2700 1 -- 172.16.96.201:6801/3074 --> osd.41 172.16.96.203:6806/2619 -- osd_sub_op(client.4133.0:104 3.3f d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 [] v 21'1 snapset=0=[]:[] snapc=0=[]) v10 -- ?+4195211 0x777d600 -32> 2014-05-19 18:38:43.028109 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] append_log log((0'0,0'0], crt=0'0) [21'1 (0'0) modify d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 by client.4133.0:104 2014-05-19 12:37:55.600025] -31> 2014-05-19 18:38:43.028140 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] add_log_entry 21'1 (0'0) modify d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 by client.4133.0:104 2014-05-19 12:37:55.600025 -30> 2014-05-19 18:38:43.028168 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] append_log adding 1 keys -29> 2014-05-19 18:38:43.028216 7fd03e6b2700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 21'1, trimmed: -28> 2014-05-19 18:38:43.028244 7fd03e6b2700 5 queue_transactions existing osr(3.3f 0x610ceb0)/0x610ceb0 -27> 2014-05-19 18:38:43.028249 7fd03e6b2700 10 op_submit_start 24886 -26> 2014-05-19 18:38:43.028251 7fd03e6b2700 5 queue_transactions (trailing journal) 24886 -25> 2014-05-19 18:38:43.028253 7fd03e6b2700 5 queue_op 0x40120c0 seq 24886 osr(3.3f 0x610ceb0) 4196755 bytes (queue has 1 ops and 4196755 bytes) -24> 2014-05-19 18:38:43.028261 7fd03e6b2700 10 op_submit_finish 24886 -23> 2014-05-19 18:38:43.028271 7fd04e104700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x400de60 (1 active) -22> 2014-05-19 18:38:43.028264 7fd03e6b2700 10 osd.0 pg_epoch: 21 pg[3.3f( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,41] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] eval_repop repgather(0x30c4a80 21'1 rep_tid=1 committed?=0 applied?=0 lock=0 op=osd_op(client.4133.0:104 rbd_data.10236b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.d310c43f ack+ondisk+write e21) v4) wants=ad -21> 2014-05-19 18:38:43.028286 7fd04e104700 5 _do_op 0x40120c0 seq 24886 osr(3.3f 0x610ceb0)/0x610ceb0 start -20> 2014-05-19 18:38:43.028290 7fd03e6b2700 10 osd.0 21 dequeue_op 0x5aaf2c0 finish -19> 2014-05-19 18:38:43.028294 7fd04e104700 10 _do_transaction on 0x594c3c0 -18> 2014-05-19 18:38:43.028296 7fd03e6b2700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active) -17> 2014-05-19 18:38:43.028300 7fd03e6b2700 20 OSD::op_tp worker waiting -16> 2014-05-19 18:38:43.028306 7fd04e104700 15 _omap_setkeys meta/51683abc/pglog_3.3f/0//-1 -15> 2014-05-19 18:38:43.028412 7fd04e104700 10 lookup_strip_header done cid meta oid 51683abc/pglog_3.3f/0//-1 -14> 2014-05-19 18:38:43.028452 7fd04e104700 10 _collection_setattr 3.3f_head 'info' len 1 -13> 2014-05-19 18:38:43.028544 7fd04e104700 10 lookup_strip_header done cid COLLECTIONS oid ecfe0533/3.3f_head/head//-1 -12> 2014-05-19 18:38:43.028586 7fd04e104700 10 _collection_setattr 3.3f_head 'info' len 1 = 0 -11> 2014-05-19 18:38:43.028603 7fd04e104700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10> 2014-05-19 18:38:43.028685 7fd04e104700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9> 2014-05-19 18:38:43.028728 7fd04e104700 15 _omap_rmkeys meta/51683abc/pglog_3.3f/0//-1 -8> 2014-05-19 18:38:43.028742 7fd04e104700 10 _omap_rmkeys meta/51683abc/pglog_3.3f/0//-1 r = 0 -7> 2014-05-19 18:38:43.028756 7fd04e104700 15 _omap_setkeys meta/51683abc/pglog_3.3f/0//-1 -6> 2014-05-19 18:38:43.028783 7fd04e104700 15 _touch 3.3f_head/d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 -5> 2014-05-19 18:38:43.028846 7fd04e104700 20 lookup_strip_header failed to get strip_header cid 3.3f_head oid d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 -4> 2014-05-19 18:38:43.028901 7fd04e104700 20 write_state seq is 8794 -3> 2014-05-19 18:38:43.028918 7fd04e104700 20 set_header setting 8793 cid 3.3f_head oid d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 parent seq 0 -2> 2014-05-19 18:38:43.028950 7fd04e104700 10 _touch 3.3f_head/d310c43f/rbd_data.10236b8b4567.0000000000000004/head//3 = 0 -1> 2014-05-19 18:38:43.028958 7fd04e104700 -1 bad op 2307 0> 2014-05-19 18:38:43.032664 7fd04e104700 -1 os/KeyValueStore.cc: In function 'unsigned int KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)' thread 7fd04e104700 time 2014-05-19 18:38:43.028966 os/KeyValueStore.cc: 1457: FAILED assert(0) ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74) 1: (KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)+0x1d0) [0x9d8720] 2: (KeyValueStore::_do_transactions(std::list >&, unsigned long, ThreadPool::TPHandle*)+0x8e) [0x9daa0e] 3: (KeyValueStore::_do_op(KeyValueStore::OpSequencer*, ThreadPool::TPHandle&)+0x97) [0x9dab17] 4: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0xb5101a] 5: (ThreadPool::WorkThread::entry()+0x10) [0xb52270] 6: (()+0x7e9a) [0x7fd056078e9a] 7: (clone()+0x6d) [0x7fd054623ccd] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 20/20 osd 20/20 optracker 0/ 5 objclass 20/20 filestore 20/20 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 20/20 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/osd.0.log --- end dump of recent events --- 2014-05-19 18:38:43.081210 7fd04aecb700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:43.081213 7fd04aecb700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:43.081215 7fd04aecb700 20 osd.0 21 _dispatch 0x66d2b40 osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 2014-05-19 18:38:43.081222 7fd04aecb700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.067403, event: header_read, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 2014-05-19 18:38:43.081234 7fd04aecb700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.067407, event: throttled, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 2014-05-19 18:38:43.081242 7fd04aecb700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.081150, event: all_read, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 2014-05-19 18:38:43.081249 7fd04aecb700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.081208, event: dispatched, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 2014-05-19 18:38:43.081257 7fd04aecb700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.081257, event: waiting_for_osdmap, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 2014-05-19 18:38:43.081411 7fd04aecb700 15 osd.0 21 require_same_or_newer_map 21 (i am 21) 0x66d2b40 2014-05-19 18:38:43.081556 7fd04aecb700 20 osd.0 21 _share_map_incoming client.4133 172.16.96.219:0/1052969 21 2014-05-19 18:38:43.081582 7fd04aecb700 15 osd.0 21 enqueue_op 0x5aaf860 prio 63 cost 4194304 latency 0.014178 osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 2014-05-19 18:38:43.081604 7fd04aecb700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:43.081606 7fd04aecb700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:43.081620 7fd03bead700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active) 2014-05-19 18:38:43.081629 7fd03bead700 10 osd.0 21 dequeue_op 0x5aaf860 prio 63 cost 4194304 latency 0.014226 osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 pg pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] 2014-05-19 18:38:43.081651 7fd03bead700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.081651, event: reached_pg, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 2014-05-19 18:38:43.081665 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-05-19 18:38:43.081680 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] handle_message: 0x5aaf860 2014-05-19 18:38:43.081690 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_op osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 may_write -> write-ordered flags ack+ondisk+write 2014-05-19 18:38:43.081716 7fd03bead700 15 getattr 3.be_head/4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 '_' 2014-05-19 18:38:43.081782 7fd03bead700 10 getattr got.size() is 0 2014-05-19 18:38:43.081785 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: no obc for soid 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 but can_create 2014-05-19 18:38:43.081797 7fd03bead700 15 getattr 3.be_head/4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 'snapset' 2014-05-19 18:38:43.081818 7fd03bead700 10 getattr got.size() is 0 2014-05-19 18:38:43.081821 7fd03bead700 15 getattr 3.be_head/4352e4be/rbd_data.10236b8b4567.000000000000000a/snapdir//3 'snapset' 2014-05-19 18:38:43.081841 7fd03bead700 10 getattr got.size() is 0 2014-05-19 18:38:43.081852 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] create_object_context 0x67458c0 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 2014-05-19 18:38:43.081862 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] populate_obc_watchers 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 2014-05-19 18:38:43.081872 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 2014-05-19 18:38:43.081881 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: 0x67458c0 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 rwstate(none n=0 w=0) oi: 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x2de9260 snapset: 0=[]:[] 2014-05-19 18:38:43.081896 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] find_object_context 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 @head oi=4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-05-19 18:38:43.081915 7fd03bead700 15 getattr 3.be_head/4352e4be/rbd_data.10236b8b4567.000000000000000a/snapdir//3 '_' 2014-05-19 18:38:43.081938 7fd03bead700 10 getattr got.size() is 0 2014-05-19 18:38:43.081940 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: no obc for soid 4352e4be/rbd_data.10236b8b4567.000000000000000a/snapdir//3 and !can_create 2014-05-19 18:38:43.081953 7fd03bead700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.081953, event: started, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 2014-05-19 18:38:43.081984 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] execute_ctx 0x785f800 2014-05-19 18:38:43.081996 7fd03bead700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.081996, event: started, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 2014-05-19 18:38:43.082013 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_op 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] ov 0'0 av 21'1 snapc 0=[] snapset 0=[]:[] 2014-05-19 18:38:43.082027 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 2014-05-19 18:38:43.082038 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-05-19 18:38:43.082054 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op write 0~4194304 2014-05-19 18:38:43.082090 7fd03bead700 15 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op_effects on session 0x2de9a40 2014-05-19 18:38:43.082102 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] make_writeable 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 snapset=0x2de9298 snapc=0=[] 2014-05-19 18:38:43.082112 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] setting DIRTY flag 2014-05-19 18:38:43.082121 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] make_writeable 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 done, snapset=0=[]:[]+head 2014-05-19 18:38:43.082131 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] finish_ctx 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 0x785f800 op modify 2014-05-19 18:38:43.082142 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] set mtime to 2014-05-19 12:37:55.657031 2014-05-19 18:38:43.082166 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] final snapset 0=[]:[]+head in 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 2014-05-19 18:38:43.082184 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] zeroing write result code 0 2014-05-19 18:38:43.082194 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] new_repop rep_tid 2 on osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 2014-05-19 18:38:43.082208 7fd03bead700 7 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] issue_repop rep_tid 2 o 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 2014-05-19 18:38:43.082227 7fd03bead700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.082227, event: waiting for subops from 40, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 2014-05-19 18:38:43.082256 7fd03bead700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 2014-05-19 18:38:43.082277 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] append_log log((0'0,0'0], crt=0'0) [21'1 (0'0) modify 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 by client.4133.0:112 2014-05-19 12:37:55.657031] 2014-05-19 18:38:43.082298 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] add_log_entry 21'1 (0'0) modify 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 by client.4133.0:112 2014-05-19 12:37:55.657031 2014-05-19 18:38:43.082319 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] append_log adding 1 keys 2014-05-19 18:38:43.082357 7fd03bead700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 21'1, trimmed: 2014-05-19 18:38:43.082376 7fd03bead700 5 queue_transactions existing osr(3.be 0x48bf360)/0x48bf360 2014-05-19 18:38:43.082381 7fd03bead700 10 op_submit_start 24887 2014-05-19 18:38:43.082382 7fd03bead700 5 queue_transactions (trailing journal) 24887 2014-05-19 18:38:43.082384 7fd03bead700 5 queue_op 0x5d48d20 seq 24887 osr(3.be 0x48bf360) 4196755 bytes (queue has 2 ops and 8393510 bytes) 2014-05-19 18:38:43.082390 7fd03bead700 10 op_submit_finish 24887 2014-05-19 18:38:43.082393 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] eval_repop repgather(0x59a1ec0 21'1 rep_tid=2 committed?=0 applied?=0 lock=0 op=osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4) wants=ad 2014-05-19 18:38:43.082416 7fd03bead700 10 osd.0 21 dequeue_op 0x5aaf860 finish 2014-05-19 18:38:43.082420 7fd03bead700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active) 2014-05-19 18:38:43.082423 7fd03bead700 20 OSD::op_tp worker waiting 2014-05-19 18:38:43.084315 7fd04e905700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3573d40 (2 active) 2014-05-19 18:38:43.084325 7fd04e905700 5 _do_op 0x5d48d20 seq 24887 osr(3.be 0x48bf360)/0x48bf360 start 2014-05-19 18:38:43.084329 7fd04e905700 10 _do_transaction on 0x596b140 2014-05-19 18:38:43.084335 7fd04e905700 15 _omap_setkeys meta/516e9ec7/pglog_3.be/0//-1 2014-05-19 18:38:43.084400 7fd04e905700 10 lookup_strip_header done cid meta oid 516e9ec7/pglog_3.be/0//-1 2014-05-19 18:38:43.084416 7fd04e905700 10 _collection_setattr 3.be_head 'info' len 1 2014-05-19 18:38:43.084461 7fd04e905700 10 lookup_strip_header done cid COLLECTIONS oid 5022dca8/3.be_head/head//-1 2014-05-19 18:38:43.084475 7fd04e905700 10 _collection_setattr 3.be_head 'info' len 1 = 0 2014-05-19 18:38:43.084481 7fd04e905700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-05-19 18:38:43.090838 7fd04aecb700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:43.090841 7fd04aecb700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:43.090843 7fd04aecb700 20 osd.0 21 _dispatch 0x5803440 osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 2014-05-19 18:38:43.090852 7fd04aecb700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.081190, event: header_read, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 2014-05-19 18:38:43.090864 7fd04aecb700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.081193, event: throttled, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 2014-05-19 18:38:43.090873 7fd04aecb700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.090764, event: all_read, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 2014-05-19 18:38:43.090882 7fd04aecb700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.090835, event: dispatched, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 2014-05-19 18:38:43.090891 7fd04aecb700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.090891, event: waiting_for_osdmap, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 2014-05-19 18:38:43.090904 7fd04aecb700 15 osd.0 21 require_same_or_newer_map 21 (i am 21) 0x5803440 2014-05-19 18:38:43.090923 7fd04aecb700 20 osd.0 21 _share_map_incoming client.4133 172.16.96.219:0/1052969 21 2014-05-19 18:38:43.090941 7fd04aecb700 15 osd.0 21 enqueue_op 0x5aaec30 prio 63 cost 4194304 latency 0.009751 osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 2014-05-19 18:38:43.090962 7fd04aecb700 10 osd.0 21 do_waiters -- start 2014-05-19 18:38:43.090964 7fd04aecb700 10 osd.0 21 do_waiters -- finish 2014-05-19 18:38:43.090975 7fd041eb9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active) 2014-05-19 18:38:43.090984 7fd041eb9700 10 osd.0 21 dequeue_op 0x5aaec30 prio 63 cost 4194304 latency 0.009794 osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 pg pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] 2014-05-19 18:38:43.090999 7fd041eb9700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.090999, event: reached_pg, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 2014-05-19 18:38:43.091013 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-05-19 18:38:43.091023 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] handle_message: 0x5aaec30 2014-05-19 18:38:43.091029 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_op osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 may_write -> write-ordered flags ack+ondisk+write 2014-05-19 18:38:43.091047 7fd041eb9700 15 getattr 3.7e_head/2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 '_' 2014-05-19 18:38:43.091104 7fd041eb9700 10 getattr got.size() is 0 2014-05-19 18:38:43.091106 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: no obc for soid 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 but can_create 2014-05-19 18:38:43.091113 7fd041eb9700 15 getattr 3.7e_head/2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 'snapset' 2014-05-19 18:38:43.091125 7fd041eb9700 10 getattr got.size() is 0 2014-05-19 18:38:43.091126 7fd041eb9700 15 getattr 3.7e_head/2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/snapdir//3 'snapset' 2014-05-19 18:38:43.091137 7fd041eb9700 10 getattr got.size() is 0 2014-05-19 18:38:43.091144 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] create_object_context 0x7365080 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 2014-05-19 18:38:43.091151 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] populate_obc_watchers 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 2014-05-19 18:38:43.091156 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 2014-05-19 18:38:43.091160 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: 0x7365080 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 rwstate(none n=0 w=0) oi: 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x2de96c0 snapset: 0=[]:[] 2014-05-19 18:38:43.091169 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] find_object_context 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 @head oi=2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-05-19 18:38:43.091181 7fd041eb9700 15 getattr 3.7e_head/2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/snapdir//3 '_' 2014-05-19 18:38:43.091194 7fd041eb9700 10 getattr got.size() is 0 2014-05-19 18:38:43.091195 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: no obc for soid 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/snapdir//3 and !can_create 2014-05-19 18:38:43.091202 7fd041eb9700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.091202, event: started, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 2014-05-19 18:38:43.091210 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] execute_ctx 0x7ad2a00 2014-05-19 18:38:43.091216 7fd041eb9700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.091216, event: started, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 2014-05-19 18:38:43.091222 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_op 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] ov 0'0 av 21'1 snapc 0=[] snapset 0=[]:[] 2014-05-19 18:38:43.091230 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 2014-05-19 18:38:43.091236 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-05-19 18:38:43.091246 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op write 0~4194304 2014-05-19 18:38:43.091256 7fd041eb9700 15 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op_effects on session 0x2de9a40 2014-05-19 18:38:43.091261 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] make_writeable 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 snapset=0x2de96f8 snapc=0=[] 2014-05-19 18:38:43.091267 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] setting DIRTY flag 2014-05-19 18:38:43.091273 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] make_writeable 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 done, snapset=0=[]:[]+head 2014-05-19 18:38:43.091278 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] finish_ctx 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 0x7ad2a00 op modify 2014-05-19 18:38:43.091284 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] set mtime to 2014-05-19 12:37:55.667445 2014-05-19 18:38:43.091299 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] final snapset 0=[]:[]+head in 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 2014-05-19 18:38:43.091312 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] zeroing write result code 0 2014-05-19 18:38:43.091317 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] new_repop rep_tid 3 on osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 2014-05-19 18:38:43.091327 7fd041eb9700 7 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] issue_repop rep_tid 3 o 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 2014-05-19 18:38:43.091340 7fd041eb9700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.091340, event: waiting for subops from 20, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 2014-05-19 18:38:43.091360 7fd041eb9700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 2014-05-19 18:38:43.091375 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] append_log log((0'0,0'0], crt=0'0) [21'1 (0'0) modify 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 by client.4133.0:114 2014-05-19 12:37:55.667445] 2014-05-19 18:38:43.091396 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] add_log_entry 21'1 (0'0) modify 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 by client.4133.0:114 2014-05-19 12:37:55.667445 2014-05-19 18:38:43.091415 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] append_log adding 1 keys 2014-05-19 18:38:43.091450 7fd041eb9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 21'1, trimmed: 2014-05-19 18:38:43.091469 7fd041eb9700 5 queue_transactions existing osr(3.7e 0x44ebae0)/0x44ebae0 2014-05-19 18:38:43.091473 7fd041eb9700 10 op_submit_start 24888 2014-05-19 18:38:43.091474 7fd041eb9700 5 queue_transactions (trailing journal) 24888 2014-05-19 18:38:43.091476 7fd041eb9700 5 queue_op 0x407cde0 seq 24888 osr(3.7e 0x44ebae0) 4196755 bytes (queue has 3 ops and 12590265 bytes) 2014-05-19 18:38:43.091478 7fd041eb9700 10 op_submit_finish 24888 2014-05-19 18:38:43.091480 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] eval_repop repgather(0x79dd200 21'1 rep_tid=3 committed?=0 applied?=0 lock=0 op=osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4) wants=ad 2014-05-19 18:38:43.091496 7fd041eb9700 10 osd.0 21 dequeue_op 0x5aaec30 finish 2014-05-19 18:38:43.091499 7fd041eb9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active) 2014-05-19 18:38:43.091501 7fd041eb9700 20 OSD::op_tp worker waiting 2014-05-19 18:38:43.095310 7fd04e104700 -1 *** Caught signal (Aborted) ** in thread 7fd04e104700 ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74) 1: /usr/local/bin/ceph-osd() [0xa85780] 2: (()+0xfcb0) [0x7fd056080cb0] 3: (gsignal()+0x35) [0x7fd054566425] 4: (abort()+0x17b) [0x7fd054569b8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fd054e61e2d] 6: (()+0x5ef26) [0x7fd054e5ff26] 7: (()+0x5ef53) [0x7fd054e5ff53] 8: (()+0x5f17e) [0x7fd054e6017e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0xb5f43d] 10: (KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)+0x1d0) [0x9d8720] 11: (KeyValueStore::_do_transactions(std::list >&, unsigned long, ThreadPool::TPHandle*)+0x8e) [0x9daa0e] 12: (KeyValueStore::_do_op(KeyValueStore::OpSequencer*, ThreadPool::TPHandle&)+0x97) [0x9dab17] 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0xb5101a] 14: (ThreadPool::WorkThread::entry()+0x10) [0xb52270] 15: (()+0x7e9a) [0x7fd056078e9a] 16: (clone()+0x6d) [0x7fd054623ccd] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -145> 2014-05-19 18:38:43.081174 7fd04aecb700 1 -- 172.16.96.201:6800/3074 <== client.4133 172.16.96.219:0/1052969 2 ==== osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 ==== 227+0+4194304 (1990630917 0 0) 0x66d2b40 con 0x61fba20 -144> 2014-05-19 18:38:43.081210 7fd04aecb700 10 osd.0 21 do_waiters -- start -143> 2014-05-19 18:38:43.081213 7fd04aecb700 10 osd.0 21 do_waiters -- finish -142> 2014-05-19 18:38:43.081215 7fd04aecb700 20 osd.0 21 _dispatch 0x66d2b40 osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 -141> 2014-05-19 18:38:43.081222 7fd04aecb700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.067403, event: header_read, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 -140> 2014-05-19 18:38:43.081234 7fd04aecb700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.067407, event: throttled, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 -139> 2014-05-19 18:38:43.081242 7fd04aecb700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.081150, event: all_read, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 -138> 2014-05-19 18:38:43.081249 7fd04aecb700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.081208, event: dispatched, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 -137> 2014-05-19 18:38:43.081257 7fd04aecb700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.081257, event: waiting_for_osdmap, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 -136> 2014-05-19 18:38:43.081411 7fd04aecb700 15 osd.0 21 require_same_or_newer_map 21 (i am 21) 0x66d2b40 -135> 2014-05-19 18:38:43.081556 7fd04aecb700 20 osd.0 21 _share_map_incoming client.4133 172.16.96.219:0/1052969 21 -134> 2014-05-19 18:38:43.081582 7fd04aecb700 15 osd.0 21 enqueue_op 0x5aaf860 prio 63 cost 4194304 latency 0.014178 osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 -133> 2014-05-19 18:38:43.081604 7fd04aecb700 10 osd.0 21 do_waiters -- start -132> 2014-05-19 18:38:43.081606 7fd04aecb700 10 osd.0 21 do_waiters -- finish -131> 2014-05-19 18:38:43.081620 7fd03bead700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active) -130> 2014-05-19 18:38:43.081629 7fd03bead700 10 osd.0 21 dequeue_op 0x5aaf860 prio 63 cost 4194304 latency 0.014226 osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 pg pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] -129> 2014-05-19 18:38:43.081651 7fd03bead700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.081651, event: reached_pg, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 -128> 2014-05-19 18:38:43.081665 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -127> 2014-05-19 18:38:43.081680 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] handle_message: 0x5aaf860 -126> 2014-05-19 18:38:43.081690 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_op osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 may_write -> write-ordered flags ack+ondisk+write -125> 2014-05-19 18:38:43.081716 7fd03bead700 15 getattr 3.be_head/4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 '_' -124> 2014-05-19 18:38:43.081782 7fd03bead700 10 getattr got.size() is 0 -123> 2014-05-19 18:38:43.081785 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: no obc for soid 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 but can_create -122> 2014-05-19 18:38:43.081797 7fd03bead700 15 getattr 3.be_head/4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 'snapset' -121> 2014-05-19 18:38:43.081818 7fd03bead700 10 getattr got.size() is 0 -120> 2014-05-19 18:38:43.081821 7fd03bead700 15 getattr 3.be_head/4352e4be/rbd_data.10236b8b4567.000000000000000a/snapdir//3 'snapset' -119> 2014-05-19 18:38:43.081841 7fd03bead700 10 getattr got.size() is 0 -118> 2014-05-19 18:38:43.081852 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] create_object_context 0x67458c0 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 -117> 2014-05-19 18:38:43.081862 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] populate_obc_watchers 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 -116> 2014-05-19 18:38:43.081872 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 -115> 2014-05-19 18:38:43.081881 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: 0x67458c0 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 rwstate(none n=0 w=0) oi: 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x2de9260 snapset: 0=[]:[] -114> 2014-05-19 18:38:43.081896 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] find_object_context 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 @head oi=4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) -113> 2014-05-19 18:38:43.081915 7fd03bead700 15 getattr 3.be_head/4352e4be/rbd_data.10236b8b4567.000000000000000a/snapdir//3 '_' -112> 2014-05-19 18:38:43.081938 7fd03bead700 10 getattr got.size() is 0 -111> 2014-05-19 18:38:43.081940 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: no obc for soid 4352e4be/rbd_data.10236b8b4567.000000000000000a/snapdir//3 and !can_create -110> 2014-05-19 18:38:43.081953 7fd03bead700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.081953, event: started, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 -109> 2014-05-19 18:38:43.081984 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] execute_ctx 0x785f800 -108> 2014-05-19 18:38:43.081996 7fd03bead700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.081996, event: started, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 -107> 2014-05-19 18:38:43.082013 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_op 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] ov 0'0 av 21'1 snapc 0=[] snapset 0=[]:[] -106> 2014-05-19 18:38:43.082027 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] -105> 2014-05-19 18:38:43.082038 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 -104> 2014-05-19 18:38:43.082054 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op write 0~4194304 -103> 2014-05-19 18:38:43.082090 7fd03bead700 15 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op_effects on session 0x2de9a40 -102> 2014-05-19 18:38:43.082102 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] make_writeable 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 snapset=0x2de9298 snapc=0=[] -101> 2014-05-19 18:38:43.082112 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] setting DIRTY flag -100> 2014-05-19 18:38:43.082121 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] make_writeable 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 done, snapset=0=[]:[]+head -99> 2014-05-19 18:38:43.082131 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] finish_ctx 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 0x785f800 op modify -98> 2014-05-19 18:38:43.082142 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] set mtime to 2014-05-19 12:37:55.657031 -97> 2014-05-19 18:38:43.082166 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] final snapset 0=[]:[]+head in 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 -96> 2014-05-19 18:38:43.082184 7fd03bead700 20 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] zeroing write result code 0 -95> 2014-05-19 18:38:43.082194 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] new_repop rep_tid 2 on osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 -94> 2014-05-19 18:38:43.082208 7fd03bead700 7 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] issue_repop rep_tid 2 o 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 -93> 2014-05-19 18:38:43.082227 7fd03bead700 5 -- op tracker -- , seq: 23063, time: 2014-05-19 18:38:43.082227, event: waiting for subops from 40, request: osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4 -92> 2014-05-19 18:38:43.082256 7fd03bead700 20 osd.0 21 _share_map_outgoing 0x3bb7b80 already has epoch 21 -91> 2014-05-19 18:38:43.082260 7fd03bead700 1 -- 172.16.96.201:6801/3074 --> osd.40 172.16.96.203:6801/2353 -- osd_sub_op(client.4133.0:112 3.be 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 [] v 21'1 snapset=0=[]:[] snapc=0=[]) v10 -- ?+4195211 0x777e100 -90> 2014-05-19 18:38:43.082277 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] append_log log((0'0,0'0], crt=0'0) [21'1 (0'0) modify 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 by client.4133.0:112 2014-05-19 12:37:55.657031] -89> 2014-05-19 18:38:43.082298 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] add_log_entry 21'1 (0'0) modify 4352e4be/rbd_data.10236b8b4567.000000000000000a/head//3 by client.4133.0:112 2014-05-19 12:37:55.657031 -88> 2014-05-19 18:38:43.082319 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] append_log adding 1 keys -87> 2014-05-19 18:38:43.082357 7fd03bead700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 21'1, trimmed: -86> 2014-05-19 18:38:43.082376 7fd03bead700 5 queue_transactions existing osr(3.be 0x48bf360)/0x48bf360 -85> 2014-05-19 18:38:43.082381 7fd03bead700 10 op_submit_start 24887 -84> 2014-05-19 18:38:43.082382 7fd03bead700 5 queue_transactions (trailing journal) 24887 -83> 2014-05-19 18:38:43.082384 7fd03bead700 5 queue_op 0x5d48d20 seq 24887 osr(3.be 0x48bf360) 4196755 bytes (queue has 2 ops and 8393510 bytes) -82> 2014-05-19 18:38:43.082390 7fd03bead700 10 op_submit_finish 24887 -81> 2014-05-19 18:38:43.082393 7fd03bead700 10 osd.0 pg_epoch: 21 pg[3.be( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,40] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] eval_repop repgather(0x59a1ec0 21'1 rep_tid=2 committed?=0 applied?=0 lock=0 op=osd_op(client.4133.0:112 rbd_data.10236b8b4567.000000000000000a [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.4352e4be ack+ondisk+write e21) v4) wants=ad -80> 2014-05-19 18:38:43.082416 7fd03bead700 10 osd.0 21 dequeue_op 0x5aaf860 finish -79> 2014-05-19 18:38:43.082420 7fd03bead700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active) -78> 2014-05-19 18:38:43.082423 7fd03bead700 20 OSD::op_tp worker waiting -77> 2014-05-19 18:38:43.084315 7fd04e905700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3573d40 (2 active) -76> 2014-05-19 18:38:43.084325 7fd04e905700 5 _do_op 0x5d48d20 seq 24887 osr(3.be 0x48bf360)/0x48bf360 start -75> 2014-05-19 18:38:43.084329 7fd04e905700 10 _do_transaction on 0x596b140 -74> 2014-05-19 18:38:43.084335 7fd04e905700 15 _omap_setkeys meta/516e9ec7/pglog_3.be/0//-1 -73> 2014-05-19 18:38:43.084400 7fd04e905700 10 lookup_strip_header done cid meta oid 516e9ec7/pglog_3.be/0//-1 -72> 2014-05-19 18:38:43.084416 7fd04e905700 10 _collection_setattr 3.be_head 'info' len 1 -71> 2014-05-19 18:38:43.084461 7fd04e905700 10 lookup_strip_header done cid COLLECTIONS oid 5022dca8/3.be_head/head//-1 -70> 2014-05-19 18:38:43.084475 7fd04e905700 10 _collection_setattr 3.be_head 'info' len 1 = 0 -69> 2014-05-19 18:38:43.084481 7fd04e905700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -68> 2014-05-19 18:38:43.090791 7fd04aecb700 1 -- 172.16.96.201:6800/3074 <== client.4133 172.16.96.219:0/1052969 3 ==== osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 ==== 227+0+4194304 (3626794993 0 0) 0x5803440 con 0x61fba20 -67> 2014-05-19 18:38:43.090838 7fd04aecb700 10 osd.0 21 do_waiters -- start -66> 2014-05-19 18:38:43.090841 7fd04aecb700 10 osd.0 21 do_waiters -- finish -65> 2014-05-19 18:38:43.090843 7fd04aecb700 20 osd.0 21 _dispatch 0x5803440 osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 -64> 2014-05-19 18:38:43.090852 7fd04aecb700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.081190, event: header_read, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 -63> 2014-05-19 18:38:43.090864 7fd04aecb700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.081193, event: throttled, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 -62> 2014-05-19 18:38:43.090873 7fd04aecb700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.090764, event: all_read, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 -61> 2014-05-19 18:38:43.090882 7fd04aecb700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.090835, event: dispatched, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 -60> 2014-05-19 18:38:43.090891 7fd04aecb700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.090891, event: waiting_for_osdmap, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 -59> 2014-05-19 18:38:43.090904 7fd04aecb700 15 osd.0 21 require_same_or_newer_map 21 (i am 21) 0x5803440 -58> 2014-05-19 18:38:43.090923 7fd04aecb700 20 osd.0 21 _share_map_incoming client.4133 172.16.96.219:0/1052969 21 -57> 2014-05-19 18:38:43.090941 7fd04aecb700 15 osd.0 21 enqueue_op 0x5aaec30 prio 63 cost 4194304 latency 0.009751 osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 -56> 2014-05-19 18:38:43.090962 7fd04aecb700 10 osd.0 21 do_waiters -- start -55> 2014-05-19 18:38:43.090964 7fd04aecb700 10 osd.0 21 do_waiters -- finish -54> 2014-05-19 18:38:43.090975 7fd041eb9700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (1 active) -53> 2014-05-19 18:38:43.090984 7fd041eb9700 10 osd.0 21 dequeue_op 0x5aaec30 prio 63 cost 4194304 latency 0.009794 osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 pg pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] -52> 2014-05-19 18:38:43.090999 7fd041eb9700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.090999, event: reached_pg, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 -51> 2014-05-19 18:38:43.091013 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -50> 2014-05-19 18:38:43.091023 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] handle_message: 0x5aaec30 -49> 2014-05-19 18:38:43.091029 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_op osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 may_write -> write-ordered flags ack+ondisk+write -48> 2014-05-19 18:38:43.091047 7fd041eb9700 15 getattr 3.7e_head/2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 '_' -47> 2014-05-19 18:38:43.091104 7fd041eb9700 10 getattr got.size() is 0 -46> 2014-05-19 18:38:43.091106 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: no obc for soid 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 but can_create -45> 2014-05-19 18:38:43.091113 7fd041eb9700 15 getattr 3.7e_head/2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 'snapset' -44> 2014-05-19 18:38:43.091125 7fd041eb9700 10 getattr got.size() is 0 -43> 2014-05-19 18:38:43.091126 7fd041eb9700 15 getattr 3.7e_head/2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/snapdir//3 'snapset' -42> 2014-05-19 18:38:43.091137 7fd041eb9700 10 getattr got.size() is 0 -41> 2014-05-19 18:38:43.091144 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] create_object_context 0x7365080 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 -40> 2014-05-19 18:38:43.091151 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] populate_obc_watchers 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 -39> 2014-05-19 18:38:43.091156 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] ReplicatedPG::check_blacklisted_obc_watchers for obc 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 -38> 2014-05-19 18:38:43.091160 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: 0x7365080 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 rwstate(none n=0 w=0) oi: 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x2de96c0 snapset: 0=[]:[] -37> 2014-05-19 18:38:43.091169 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] find_object_context 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 @head oi=2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) -36> 2014-05-19 18:38:43.091181 7fd041eb9700 15 getattr 3.7e_head/2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/snapdir//3 '_' -35> 2014-05-19 18:38:43.091194 7fd041eb9700 10 getattr got.size() is 0 -34> 2014-05-19 18:38:43.091195 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] get_object_context: no obc for soid 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/snapdir//3 and !can_create -33> 2014-05-19 18:38:43.091202 7fd041eb9700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.091202, event: started, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 -32> 2014-05-19 18:38:43.091210 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] execute_ctx 0x7ad2a00 -31> 2014-05-19 18:38:43.091216 7fd041eb9700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.091216, event: started, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 -30> 2014-05-19 18:38:43.091222 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_op 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] ov 0'0 av 21'1 snapc 0=[] snapset 0=[]:[] -29> 2014-05-19 18:38:43.091230 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] -28> 2014-05-19 18:38:43.091236 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 -27> 2014-05-19 18:38:43.091246 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op write 0~4194304 -26> 2014-05-19 18:38:43.091256 7fd041eb9700 15 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] do_osd_op_effects on session 0x2de9a40 -25> 2014-05-19 18:38:43.091261 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] make_writeable 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 snapset=0x2de96f8 snapc=0=[] -24> 2014-05-19 18:38:43.091267 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] setting DIRTY flag -23> 2014-05-19 18:38:43.091273 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] make_writeable 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 done, snapset=0=[]:[]+head -22> 2014-05-19 18:38:43.091278 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] finish_ctx 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 0x7ad2a00 op modify -21> 2014-05-19 18:38:43.091284 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] set mtime to 2014-05-19 12:37:55.667445 -20> 2014-05-19 18:38:43.091299 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=0 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] final snapset 0=[]:[]+head in 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 -19> 2014-05-19 18:38:43.091312 7fd041eb9700 20 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] zeroing write result code 0 -18> 2014-05-19 18:38:43.091317 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] new_repop rep_tid 3 on osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 -17> 2014-05-19 18:38:43.091327 7fd041eb9700 7 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] issue_repop rep_tid 3 o 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 -16> 2014-05-19 18:38:43.091340 7fd041eb9700 5 -- op tracker -- , seq: 23064, time: 2014-05-19 18:38:43.091340, event: waiting for subops from 20, request: osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4 -15> 2014-05-19 18:38:43.091360 7fd041eb9700 20 osd.0 21 _share_map_outgoing 0x3059e40 already has epoch 21 -14> 2014-05-19 18:38:43.091362 7fd041eb9700 1 -- 172.16.96.201:6801/3074 --> osd.20 172.16.96.202:6801/2218 -- osd_sub_op(client.4133.0:114 3.7e 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 [] v 21'1 snapset=0=[]:[] snapc=0=[]) v10 -- ?+4195211 0x777ec00 -13> 2014-05-19 18:38:43.091375 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( empty local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 crt=0'0 mlcod 0'0 active+clean] append_log log((0'0,0'0], crt=0'0) [21'1 (0'0) modify 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 by client.4133.0:114 2014-05-19 12:37:55.667445] -12> 2014-05-19 18:38:43.091396 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] add_log_entry 21'1 (0'0) modify 2bbb6c7e/rbd_data.10236b8b4567.000000000000000c/head//3 by client.4133.0:114 2014-05-19 12:37:55.667445 -11> 2014-05-19 18:38:43.091415 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] append_log adding 1 keys -10> 2014-05-19 18:38:43.091450 7fd041eb9700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 21'1, trimmed: -9> 2014-05-19 18:38:43.091469 7fd041eb9700 5 queue_transactions existing osr(3.7e 0x44ebae0)/0x44ebae0 -8> 2014-05-19 18:38:43.091473 7fd041eb9700 10 op_submit_start 24888 -7> 2014-05-19 18:38:43.091474 7fd041eb9700 5 queue_transactions (trailing journal) 24888 -6> 2014-05-19 18:38:43.091476 7fd041eb9700 5 queue_op 0x407cde0 seq 24888 osr(3.7e 0x44ebae0) 4196755 bytes (queue has 3 ops and 12590265 bytes) -5> 2014-05-19 18:38:43.091478 7fd041eb9700 10 op_submit_finish 24888 -4> 2014-05-19 18:38:43.091480 7fd041eb9700 10 osd.0 pg_epoch: 21 pg[3.7e( v 21'1 (0'0,21'1] local-les=21 n=1 ec=18 les/c 21/21 20/20/18) [0,20] r=0 lpr=20 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] eval_repop repgather(0x79dd200 21'1 rep_tid=3 committed?=0 applied?=0 lock=0 op=osd_op(client.4133.0:114 rbd_data.10236b8b4567.000000000000000c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.2bbb6c7e ack+ondisk+write e21) v4) wants=ad -3> 2014-05-19 18:38:43.091496 7fd041eb9700 10 osd.0 21 dequeue_op 0x5aaec30 finish -2> 2014-05-19 18:38:43.091499 7fd041eb9700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active) -1> 2014-05-19 18:38:43.091501 7fd041eb9700 20 OSD::op_tp worker waiting 0> 2014-05-19 18:38:43.095310 7fd04e104700 -1 *** Caught signal (Aborted) ** in thread 7fd04e104700 ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74) 1: /usr/local/bin/ceph-osd() [0xa85780] 2: (()+0xfcb0) [0x7fd056080cb0] 3: (gsignal()+0x35) [0x7fd054566425] 4: (abort()+0x17b) [0x7fd054569b8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fd054e61e2d] 6: (()+0x5ef26) [0x7fd054e5ff26] 7: (()+0x5ef53) [0x7fd054e5ff53] 8: (()+0x5f17e) [0x7fd054e6017e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0xb5f43d] 10: (KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)+0x1d0) [0x9d8720] 11: (KeyValueStore::_do_transactions(std::list >&, unsigned long, ThreadPool::TPHandle*)+0x8e) [0x9daa0e] 12: (KeyValueStore::_do_op(KeyValueStore::OpSequencer*, ThreadPool::TPHandle&)+0x97) [0x9dab17] 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0xb5101a] 14: (ThreadPool::WorkThread::entry()+0x10) [0xb52270] 15: (()+0x7e9a) [0x7fd056078e9a] 16: (clone()+0x6d) [0x7fd054623ccd] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 20/20 osd 20/20 optracker 0/ 5 objclass 20/20 filestore 20/20 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 20/20 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/osd.0.log --- end dump of recent events ---