2015-02-25 10:35:13.365602 7f0af645e800 0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578), process ceph-osd, pid 30245 2015-02-25 10:35:13.377506 7f0af645e800 0 filestore(/var/lib/ceph/osd/ceph-2) backend generic (magic 0xef53) 2015-02-25 10:35:13.386204 7f0af645e800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is supported and appears to work 2015-02-25 10:35:13.386264 7f0af645e800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-02-25 10:35:13.517642 7f0af645e800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syscall(SYS_syncfs, fd) fully supported 2015-02-25 10:35:13.524259 7f0af645e800 0 filestore(/var/lib/ceph/osd/ceph-2) limited size xattrs 2015-02-25 10:35:13.776622 7f0af645e800 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-02-25 10:35:13.788925 7f0af645e800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2015-02-25 10:35:13.788983 7f0af645e800 1 journal _open /var/lib/ceph/osd/ceph-2/journal fd 21: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0 2015-02-25 10:35:13.792260 7f0af645e800 1 journal _open /var/lib/ceph/osd/ceph-2/journal fd 21: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0 2015-02-25 10:35:13.802896 7f0af645e800 1 journal close /var/lib/ceph/osd/ceph-2/journal 2015-02-25 10:35:13.813320 7f0af645e800 0 filestore(/var/lib/ceph/osd/ceph-2) backend generic (magic 0xef53) 2015-02-25 10:35:13.851450 7f0af645e800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is supported and appears to work 2015-02-25 10:35:13.851503 7f0af645e800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-02-25 10:35:13.909483 7f0af645e800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syscall(SYS_syncfs, fd) fully supported 2015-02-25 10:35:13.918796 7f0af645e800 0 filestore(/var/lib/ceph/osd/ceph-2) limited size xattrs 2015-02-25 10:35:13.964739 7f0af645e800 0 filestore(/var/lib/ceph/osd/ceph-2) mount: WRITEAHEAD journal mode explicitly enabled in conf 2015-02-25 10:35:13.972104 7f0af645e800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2015-02-25 10:35:13.972148 7f0af645e800 1 journal _open /var/lib/ceph/osd/ceph-2/journal fd 21: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0 2015-02-25 10:35:13.973068 7f0af645e800 1 journal _open /var/lib/ceph/osd/ceph-2/journal fd 21: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0 2015-02-25 10:35:13.976401 7f0af645e800 0 cls/hello/cls_hello.cc:271: loading cls_hello 2015-02-25 10:35:13.980251 7f0af645e800 0 osd.2 33 crush map has features 1107558400, adjusting msgr requires for clients 2015-02-25 10:35:13.980297 7f0af645e800 0 osd.2 33 crush map has features 1107558400 was 8705, adjusting msgr requires for mons 2015-02-25 10:35:13.980332 7f0af645e800 0 osd.2 33 crush map has features 1107558400, adjusting msgr requires for osds 2015-02-25 10:35:13.980373 7f0af645e800 0 osd.2 33 load_pgs 2015-02-25 10:35:14.286576 7f0af645e800 0 osd.2 33 load_pgs opened 768 pgs 2015-02-25 10:35:14.288666 7f0af645e800 -1 osd.2 33 set_disk_tp_priority(22) Invalid argument: osd_disk_thread_ioprio_class is but only the following values are allowed: idle, be or rt 2015-02-25 10:35:14.293082 7f0ae2a3d700 0 osd.2 33 ignoring osdmap until we have initialized 2015-02-25 10:35:14.293316 7f0ae2a3d700 0 osd.2 33 ignoring osdmap until we have initialized 2015-02-25 10:35:14.525191 7f0ac9914700 0 -- 0.0.0.0:6805/30245 >> 172.20.32.5:6801/28932 pipe(0x3ca9340 sd=25 :56158 s=1 pgs=0 cs=0 l=0 c=0x49bb180).connect claims to be 172.20.32.5:6801/29835 not 172.20.32.5:6801/28932 - wrong node! 2015-02-25 10:35:14.525485 7f0ac9914700 0 -- 0.0.0.0:6805/30245 >> 172.20.32.5:6801/28932 pipe(0x3ca9340 sd=25 :56158 s=1 pgs=0 cs=0 l=0 c=0x49bb180).fault with nothing to send, going to standby 2015-02-25 10:35:14.884230 7f0af645e800 0 osd.2 33 done with init, starting boot process 2015-02-25 10:35:15.602414 7f0ac8b0f700 0 -- 172.20.32.100:6805/30245 >> :/0 pipe(0x3caa100 sd=70 :6805 s=0 pgs=0 cs=0 l=0 c=0x49bb860).accept failed to getpeername (107) Transport endpoint is not connected 2015-02-25 10:35:15.641687 7f0ac8e12700 0 -- 172.20.32.100:6805/30245 >> 172.20.32.17:6801/27293 pipe(0x3ca9600 sd=56 :6805 s=2 pgs=28 cs=1 l=0 c=0x49bb2e0).fault with nothing to send, going to standby 2015-02-25 10:35:31.463787 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849731.463780000 2015-02-25 10:35:31.481012 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849731.481008000 2015-02-25 10:35:31.528046 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849731.528043000 2015-02-25 10:35:31.582260 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849731.582257000 2015-02-25 10:35:31.654113 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849731.654109000 2015-02-25 10:35:31.805843 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849731.805840000 2015-02-25 10:35:31.985354 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849731.985351000 2015-02-25 10:35:32.079427 7f0ad1221700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849732.079424000 2015-02-25 10:35:32.104341 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849732.104339000 2015-02-25 10:35:32.135944 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849732.135941000 2015-02-25 10:35:32.151628 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849732.151625000 2015-02-25 10:35:32.172510 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849732.172508000 2015-02-25 10:35:32.324558 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849732.324554000 2015-02-25 10:35:32.487955 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849732.487950000 2015-02-25 10:35:32.529601 7f0ad1221700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849732.529597000 2015-02-25 10:36:33.087305 7f0ad1221700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849793.087302000 2015-02-25 10:36:33.212998 7f0ad2623700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849793.212985000 2015-02-25 10:36:33.236389 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849793.236386000 2015-02-25 10:36:33.280797 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849793.280794000 2015-02-25 10:36:33.336320 7f0ad2623700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849793.336314000 2015-02-25 10:36:35.864753 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849795.864749000 2015-02-25 10:36:38.533702 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849798.533695000 2015-02-25 10:36:38.739567 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849798.739560000 2015-02-25 10:36:38.784106 7f0ad4426700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849798.784102000 2015-02-25 10:36:38.796650 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849798.796647000 2015-02-25 10:36:38.845524 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849798.845520000 2015-02-25 10:36:38.874811 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849798.874805000 2015-02-25 10:36:38.904295 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849798.904289000 2015-02-25 10:36:38.945958 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849798.945954000 2015-02-25 10:36:38.985868 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849798.985864000 2015-02-25 10:37:32.393611 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849852.393606000 2015-02-25 10:37:32.590218 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849852.590212000 2015-02-25 10:37:32.606038 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849852.606034000 2015-02-25 10:37:32.699117 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849852.699114000 2015-02-25 10:37:32.727754 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849852.727750000 2015-02-25 10:37:32.809473 7f0ad4426700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849852.809468000 2015-02-25 10:37:32.876668 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849852.876658000 2015-02-25 10:37:32.894148 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849852.894141000 2015-02-25 10:37:32.936408 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849852.936404000 2015-02-25 10:37:32.975110 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849852.975106000 2015-02-25 10:37:33.045044 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849853.045041000 2015-02-25 10:37:33.249972 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849853.249968000 2015-02-25 10:37:33.345981 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849853.345976000 2015-02-25 10:37:33.473586 7f0ad4426700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849853.473580000 2015-02-25 10:37:33.485065 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849853.485062000 2015-02-25 10:38:32.557255 7f0ad2623700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849912.557252000 2015-02-25 10:38:32.630707 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849912.630703000 2015-02-25 10:38:34.689356 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849914.689353000 2015-02-25 10:38:40.540746 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849920.540743000 2015-02-25 10:38:40.690271 7f0ad1221700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849920.690268000 2015-02-25 10:38:40.724257 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849920.724255000 2015-02-25 10:38:40.745609 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849920.745607000 2015-02-25 10:38:40.760250 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849920.760247000 2015-02-25 10:38:40.794212 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849920.794210000 2015-02-25 10:38:40.832217 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849920.832213000 2015-02-25 10:38:40.900902 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849920.900899000 2015-02-25 10:38:40.935569 7f0ad4426700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849920.935567000 2015-02-25 10:38:41.034300 7f0ad2623700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849921.034298000 2015-02-25 10:38:41.069640 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849921.069638000 2015-02-25 10:38:41.116507 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849921.116504000 2015-02-25 10:39:33.223998 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849973.223995000 2015-02-25 10:39:33.296264 7f0ad4426700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849973.296261000 2015-02-25 10:39:33.311306 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849973.311303000 2015-02-25 10:39:33.352877 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849973.352874000 2015-02-25 10:39:33.375414 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849973.375411000 2015-02-25 10:39:33.407588 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849973.407584000 2015-02-25 10:39:33.441410 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849973.441406000 2015-02-25 10:39:34.228957 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849974.228953000 2015-02-25 10:39:34.269849 7f0ad4426700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849974.269845000 2015-02-25 10:39:34.492505 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849974.492496000 2015-02-25 10:39:36.634258 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849976.634253000 2015-02-25 10:39:36.656772 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849976.656770000 2015-02-25 10:39:36.768270 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849976.768262000 2015-02-25 10:39:36.897312 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849976.897309000 2015-02-25 10:39:36.980206 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424849976.980203000 2015-02-25 10:40:38.228128 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850038.228124000 2015-02-25 10:40:42.337829 7f0ad4426700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850042.337825000 2015-02-25 10:40:42.401998 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850042.401995000 2015-02-25 10:40:42.436206 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850042.436202000 2015-02-25 10:40:42.480414 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850042.480409000 2015-02-25 10:40:42.540722 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850042.540719000 2015-02-25 10:40:42.620394 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850042.620391000 2015-02-25 10:40:42.774928 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850042.774925000 2015-02-25 10:40:43.024872 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850043.024869000 2015-02-25 10:40:43.113902 7f0ad1221700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850043.113899000 2015-02-25 10:40:43.123961 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850043.123958000 2015-02-25 10:40:43.143732 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850043.143729000 2015-02-25 10:40:43.177264 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850043.177261000 2015-02-25 10:40:43.199636 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850043.199633000 2015-02-25 10:40:43.242148 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850043.242145000 2015-02-25 10:41:34.974638 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850094.974635000 2015-02-25 10:41:37.382912 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850097.382908000 2015-02-25 10:41:37.531864 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850097.531859000 2015-02-25 10:41:37.572099 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850097.572096000 2015-02-25 10:41:39.332847 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850099.332841000 2015-02-25 10:41:39.462597 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850099.462591000 2015-02-25 10:41:54.909659 7f0aeac84700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:34.909657) 2015-02-25 10:41:55.909940 7f0aeac84700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:35.909937) 2015-02-25 10:41:56.526526 7f0acd61b700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:36.526524) 2015-02-25 10:41:56.910112 7f0aeac84700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:36.910110) 2015-02-25 10:41:57.026830 7f0acd61b700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:37.026829) 2015-02-25 10:41:57.910334 7f0aeac84700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:37.910332) 2015-02-25 10:41:58.910576 7f0aeac84700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:38.910574) 2015-02-25 10:41:59.910795 7f0aeac84700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:39.910792) 2015-02-25 10:41:59.927036 7f0acd61b700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:39.927035) 2015-02-25 10:42:00.911035 7f0aeac84700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:40.911033) 2015-02-25 10:42:01.911284 7f0aeac84700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:41.911282) 2015-02-25 10:42:02.911575 7f0aeac84700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:42.911573) 2015-02-25 10:42:03.427328 7f0acd61b700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:43.427327) 2015-02-25 10:42:03.911724 7f0aeac84700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:43.911722) 2015-02-25 10:42:03.927631 7f0acd61b700 -1 osd.2 48 heartbeat_check: no reply from osd.1 since back 2015-02-25 10:41:33.924059 front 2015-02-25 10:41:33.924059 (cutoff 2015-02-25 10:41:43.927630) 2015-02-25 10:42:10.496634 7f0aeac84700 0 log_channel(default) log [WRN] : 15 slow requests, 15 included below; oldest blocked for > 31.057818 secs 2015-02-25 10:42:10.496640 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 30.523313 seconds old, received at 2015-02-25 10:41:39.973210: osd_op(client.4303.0:856 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_2 [write 3145728~524288] 10.41522259 ack+ondisk+write+known_if_redirected e48) currently waiting for subops from 0,1 2015-02-25 10:42:10.496644 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 30.434163 seconds old, received at 2015-02-25 10:41:40.062360: osd_op(client.4303.0:857 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_2 [write 3670016~524288] 10.41522259 ack+ondisk+write+known_if_redirected e48) currently waiting for subops from 0,1 2015-02-25 10:42:10.496648 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 30.344405 seconds old, received at 2015-02-25 10:41:40.152118: osd_op(client.4303.0:858 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_3 [writefull 0~0] 10.9b395f78 ack+ondisk+write+known_if_redirected e48) currently waiting for subops from 0,1 2015-02-25 10:42:10.496651 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 30.344095 seconds old, received at 2015-02-25 10:41:40.152428: osd_op(client.4303.0:859 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_3 [writefull 0~524288] 10.9b395f78 ack+ondisk+write+known_if_redirected e48) currently waiting for subops from 0,1 2015-02-25 10:42:10.496655 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 30.256362 seconds old, received at 2015-02-25 10:41:40.240161: osd_op(client.4303.0:860 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_3 [write 524288~524288] 10.9b395f78 ack+ondisk+write+known_if_redirected e48) currently waiting for subops from 0,1 2015-02-25 10:42:11.610690 7f0aeac84700 0 log_channel(default) log [WRN] : 17 slow requests, 12 included below; oldest blocked for > 32.171922 secs 2015-02-25 10:42:11.610694 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 31.281485 seconds old, received at 2015-02-25 10:41:40.329142: osd_op(client.4303.0:861 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_3 [write 1048576~524288] 10.9b395f78 ack+ondisk+write+known_if_redirected e48) currently reached_pg 2015-02-25 10:42:11.610713 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 32.171922 seconds old, received at 2015-02-25 10:41:39.438705: osd_op(client.4303.0:848 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_2 [writefull 0~524288] 10.41522259 ack+ondisk+write+known_if_redirected e48) currently reached_pg 2015-02-25 10:42:11.610717 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 31.237683 seconds old, received at 2015-02-25 10:41:40.372944: osd_op(client.4303.0:862 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_3 [write 1572864~524288] 10.9b395f78 ack+ondisk+write+known_if_redirected e48) currently reached_pg 2015-02-25 10:42:11.610720 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 31.193376 seconds old, received at 2015-02-25 10:41:40.417251: osd_op(client.4303.0:863 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_3 [write 2097152~524288] 10.9b395f78 ack+ondisk+write+known_if_redirected e48) currently reached_pg 2015-02-25 10:42:11.610722 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 32.146127 seconds old, received at 2015-02-25 10:41:39.464500: osd_op(client.4303.0:850 gc.11 [call lock.unlock] 4.6a0e9a15 ondisk+write+known_if_redirected e48) currently reached_pg 2015-02-25 10:42:13.133341 7f0aeac84700 0 log_channel(default) log [WRN] : 17 slow requests, 7 included below; oldest blocked for > 33.694563 secs 2015-02-25 10:42:13.133346 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 32.671866 seconds old, received at 2015-02-25 10:41:40.461402: osd_op(client.4303.0:864 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_3 [write 2621440~524288] 10.9b395f78 ack+ondisk+write+known_if_redirected e48) currently reached_pg 2015-02-25 10:42:13.133349 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 32.458615 seconds old, received at 2015-02-25 10:41:40.674653: osd_op(client.4303.0:865 usage.5 [call rgw.user_usage_log_add] 8.7e50d742 ondisk+write+known_if_redirected e48) currently reached_pg 2015-02-25 10:42:13.133354 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 33.609130 seconds old, received at 2015-02-25 10:41:39.524138: osd_op(client.4303.0:851 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_2 [write 524288~524288] 10.41522259 ack+ondisk+write+known_if_redirected e48) currently reached_pg 2015-02-25 10:42:13.133357 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 33.519077 seconds old, received at 2015-02-25 10:41:39.614191: osd_op(client.4303.0:852 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_2 [write 1048576~524288] 10.41522259 ack+ondisk+write+known_if_redirected e48) currently reached_pg 2015-02-25 10:42:13.133359 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 33.429358 seconds old, received at 2015-02-25 10:41:39.703910: osd_op(client.4303.0:853 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_2 [write 1572864~524288] 10.41522259 ack+ondisk+write+known_if_redirected e48) currently reached_pg 2015-02-25 10:42:14.548603 7f0aeac84700 0 log_channel(default) log [WRN] : 17 slow requests, 2 included below; oldest blocked for > 35.109820 secs 2015-02-25 10:42:14.548614 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 34.752798 seconds old, received at 2015-02-25 10:41:39.795727: osd_op(client.4303.0:854 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_2 [write 2097152~524288] 10.41522259 ack+ondisk+write+known_if_redirected e48) currently reached_pg 2015-02-25 10:42:14.548621 7f0aeac84700 0 log_channel(default) log [WRN] : slow request 34.664601 seconds old, received at 2015-02-25 10:41:39.883924: osd_op(client.4303.0:855 default.4303.1__shadow_putfile.2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt.2_2 [write 2621440~524288] 10.41522259 ack+ondisk+write+known_if_redirected e48) currently reached_pg 2015-02-25 10:42:20.149094 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850140.149090000 2015-02-25 10:42:20.427784 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850140.427780000 2015-02-25 10:42:20.923757 7f0ad1221700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850140.923753000 2015-02-25 10:42:21.040356 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850141.040353000 2015-02-25 10:42:21.059984 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850141.059981000 2015-02-25 10:42:21.069784 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850141.069781000 2015-02-25 10:42:21.113726 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850141.113723000 2015-02-25 10:42:21.156233 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850141.156230000 2015-02-25 10:42:21.300988 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850141.300984000 2015-02-25 10:42:21.380102 7f0ad1221700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850141.380100000 2015-02-25 10:42:34.479603 7f0ad4426700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.479600000 2015-02-25 10:42:34.496709 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.496707000 2015-02-25 10:42:34.511127 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.511124000 2015-02-25 10:42:34.554141 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.554138000 2015-02-25 10:42:34.583281 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.583277000 2015-02-25 10:42:34.609999 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.609996000 2015-02-25 10:42:34.666209 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.666207000 2015-02-25 10:42:34.727015 7f0ad1221700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.727011000 2015-02-25 10:42:34.804928 7f0ad2623700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.804925000 2015-02-25 10:42:34.813347 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.813345000 2015-02-25 10:42:34.873985 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.873982000 2015-02-25 10:42:34.887808 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.887806000 2015-02-25 10:42:34.921986 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.921984000 2015-02-25 10:42:34.944626 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.944623000 2015-02-25 10:42:34.993172 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850154.993169000 2015-02-25 10:42:35.020268 7f0ad2623700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850155.020265000 2015-02-25 10:42:35.122101 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850155.122096000 2015-02-25 10:42:35.173400 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850155.173397000 2015-02-25 10:42:35.246100 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850155.246097000 2015-02-25 10:43:35.287672 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.287662000 2015-02-25 10:43:35.303778 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.303774000 2015-02-25 10:43:35.313326 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.313323000 2015-02-25 10:43:35.323438 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.323435000 2015-02-25 10:43:35.392489 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.392484000 2015-02-25 10:43:35.404095 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.404091000 2015-02-25 10:43:35.451497 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.451493000 2015-02-25 10:43:35.502625 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.502621000 2015-02-25 10:43:35.540691 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.540688000 2015-02-25 10:43:35.620662 7f0ad1221700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.620659000 2015-02-25 10:43:35.648487 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.648485000 2015-02-25 10:43:35.682606 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.682603000 2015-02-25 10:43:35.701278 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.701275000 2015-02-25 10:43:35.716398 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.716394000 2015-02-25 10:43:35.766812 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.766809000 2015-02-25 10:43:35.816255 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.816251000 2015-02-25 10:43:35.873244 7f0ad1221700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850215.873240000 2015-02-25 10:43:36.171274 7f0ad2623700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850216.171271000 2015-02-25 10:43:36.198770 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850216.198766000 2015-02-25 10:44:36.775122 7f0ad2623700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850276.775118000 2015-02-25 10:44:36.806569 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850276.806565000 2015-02-25 10:44:36.836421 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850276.836417000 2015-02-25 10:44:36.864300 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850276.864296000 2015-02-25 10:44:36.888113 7f0ad5828700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850276.888110000 2015-02-25 10:44:36.915252 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850276.915249000 2015-02-25 10:44:36.962225 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850276.962221000 2015-02-25 10:44:36.988741 7f0ad2623700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850276.988739000 2015-02-25 10:44:37.112726 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850277.112722000 2015-02-25 10:44:37.177736 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850277.177730000 2015-02-25 10:44:37.216193 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850277.216186000 2015-02-25 10:44:37.296017 7f0ad4426700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850277.296014000 2015-02-25 10:44:37.311048 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850277.311045000 2015-02-25 10:44:37.345152 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850277.345148000 2015-02-25 10:44:37.369351 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850277.369347000 2015-02-25 10:44:37.384574 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850277.384571000 2015-02-25 10:44:37.440861 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850277.440858000 2015-02-25 10:44:37.501598 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850277.501595000 2015-02-25 10:44:37.577169 7f0ad1221700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850277.577165000 2015-02-25 10:45:36.590634 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850336.590630000 2015-02-25 10:45:36.640102 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850336.640098000 2015-02-25 10:45:36.656057 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850336.656054000 2015-02-25 10:45:36.690704 7f0ad2623700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850336.690701000 2015-02-25 10:45:36.723554 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850336.723550000 2015-02-25 10:45:36.754703 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850336.754699000 2015-02-25 10:45:36.790378 7f0ad2623700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850336.790373000 2015-02-25 10:45:36.852944 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850336.852938000 2015-02-25 10:45:36.892110 7f0ad4e27700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850336.892105000 2015-02-25 10:45:36.937199 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850336.937195000 2015-02-25 10:45:37.012431 7f0ad4426700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850337.012427000 2015-02-25 10:45:37.049764 7f0ad3a25700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850337.049759000 2015-02-25 10:45:37.070800 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850337.070797000 2015-02-25 10:45:37.098041 7f0acfe1f700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850337.098037000 2015-02-25 10:45:37.133658 7f0ad0820700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850337.133655000 2015-02-25 10:45:37.170070 7f0ad3024700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850337.170068000 2015-02-25 10:45:37.216174 7f0ad1c22700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850337.216171000 2015-02-25 10:45:37.266687 7f0ad1221700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850337.266683000 2015-02-25 10:45:39.681323 7f0ad2623700 0 cls/rgw/cls_rgw.cc:1458: gc_iterate_entries end_key=1_01424850339.681317000