2016-06-02 14:09:21.257410 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:21.257699 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:21.257705 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:21.257707 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:21.257722 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c2c0f70:23RGWListBucketIndexLogCR: operate()
2016-06-02 14:09:21.257726 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:21.257729 7f7760ff9700 20 parsed entry: id=00000000471.1088.2 iter->object=atroll5 iter->instance= name=atroll5 instance= ns=
2016-06-02 14:09:21.257733 7f7760ff9700 20 [inc sync] skipping object: lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1/atroll5: non-complete operation
2016-06-02 14:09:21.257734 7f7760ff9700 20 parsed entry: id=00000000472.1089.3 iter->object=atroll5 iter->instance= name=atroll5 instance= ns=
2016-06-02 14:09:21.257736 7f7760ff9700 20 [inc sync] syncing object: lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1/atroll5
2016-06-02 14:09:21.257740 7f7760ff9700 20 abhi: couldn't find key, no sync in progress for object atroll5
2016-06-02 14:09:21.257743 7f7760ff9700 20 operate(): entry->timestamp=2016-06-02 14:09:19.0.256194s
2016-06-02 14:09:21.257790 7f7760ff9700 20 bucket sync single entry (source_zone=91441c60-0ca1-4118-9685-990e04272142) b=lotr(@{i=us-east-1.rgw.buckets.index,e=us-east-1.rgw.buckets.non-ec}us-east-1.rgw.buckets.data[62bc922d-f295-4067-ae36-e23e2f231aad.24099.1]):-1/atroll5[0] log_entry=00000000472.1089.3 op=1 op_state=1
2016-06-02 14:09:21.257797 7f7760ff9700 20 operate(): listing bilog for incremental sync
2016-06-02 14:09:21.257811 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c2ff160:26RGWBucketSyncSingleEntryCRINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE11rgw_obj_keyE: operate()
2016-06-02 14:09:21.257819 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c2c0f70:23RGWListBucketIndexLogCR: operate()
2016-06-02 14:09:21.257824 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c2f3240:14RGWRemoveObjCR: operate()
2016-06-02 14:09:21.257825 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:21.257826 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c2f3240:14RGWRemoveObjCR: operate()
2016-06-02 14:09:21.257836 7f7760ff9700 20 enqueued request req=0x7f773c4e37c0
2016-06-02 14:09:21.257837 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:21.257838 7f7760ff9700 20 req: 0x7f773c4e37c0
2016-06-02 14:09:21.257842 7f7760ff9700 20 run: stack=0x7f773c331700 is io blocked
2016-06-02 14:09:21.257843 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:21.257857 7f77b0ff9700 20 dequeued request req=0x7f773c4e37c0
2016-06-02 14:09:21.257861 7f77b0ff9700 20 RGWWQ: empty
2016-06-02 14:09:21.257869 7f77b0ff9700  0 _send_request(): deleting obj=lotr:atroll5
2016-06-02 14:09:21.257882 7f77b0ff9700 20 get_obj_state: rctx=0x7f77b0ff6b30 obj=lotr:atroll5 state=0x7f777401f748 s->prefetch_data=0
2016-06-02 14:09:21.257903 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:21 2016
2016-06-02 14:09:21.257907 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:21 2016
2016-06-02 14:09:21.257910 7f7760ff9700 10 get_canon_resource(): dest=/admin/log
2016-06-02 14:09:21.257910 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:21 2016
/admin/log
2016-06-02 14:09:21.257922 7f77b0ff9700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2468 10.a8f56d53 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1_atroll5 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f777401d370 con 0x5640603f1eb0
2016-06-02 14:09:21.257944 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:0h71hgH744EnP4okDuv39BWLgeQ=
2016-06-02 14:09:21.257955 7f7760ff9700 20 sending request to http://localhost:8002/admin/log?bucket-instance=lotr%3A62bc922d-f295-4067-ae36-e23e2f231aad.24099.1&format=json&marker=00000000472.1089.3&type=bucket-index&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:21.257962 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=329, easy_handle=0x7f773c526e20
2016-06-02 14:09:21.257970 7f7760ff9700 20 run: stack=0x7f773c0a9c90 is io blocked
2016-06-02 14:09:21.257975 7f77477fe700 20 link_request req_data=0x7f773c065e20 req_data->id=329, easy_handle=0x7f773c526e20
2016-06-02 14:09:21.258307 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 834 ==== osd_op_reply(2468 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1_atroll5 [getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 214+0+0 (1330974727 0 0) 0x7f77fc017c30 con 0x5640603f1eb0
2016-06-02 14:09:21.258412 7f77b0ff9700 20 get_obj_state: rctx=0x7f77b0ff6b30 obj=lotr:atroll5 state=0x7f777401f748 s->prefetch_data=0
2016-06-02 14:09:21.258418 7f77b0ff9700 10 If-UnModified-Since: 2016-06-02 14:09:19.0.256194s Last-Modified: 0.000000
2016-06-02 14:09:21.258437 7f77b0ff9700 20 _send_request(): delete_obj() obj=lotr:atroll5 returned ret=-2
2016-06-02 14:09:21.258452 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c2f3240:14RGWRemoveObjCR: operate()
2016-06-02 14:09:21.258461 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c2f3240:14RGWRemoveObjCR: operate() returned r=-2
2016-06-02 14:09:21.258466 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c2ff160:26RGWBucketSyncSingleEntryCRINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE11rgw_obj_keyE: operate()
2016-06-02 14:09:21.258471 7f7760ff9700 20 abhi: deleting markeratroll5marker: 00000000472.1089.3
2016-06-02 14:09:21.258477 7f7760ff9700 20 store_marker(): updating marker marker_oid=bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 marker=00000000472.1089.3
2016-06-02 14:09:21.258485 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c028a90:26RGWSimpleRadosWriteAttrsCR: operate()
2016-06-02 14:09:21.258487 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c028a90:26RGWSimpleRadosWriteAttrsCR: operate()
2016-06-02 14:09:21.258494 7f7760ff9700 20 enqueued request req=0x7f773c0b63b0
2016-06-02 14:09:21.258496 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:21.258497 7f7760ff9700 20 req: 0x7f773c0b63b0
2016-06-02 14:09:21.258500 7f7760ff9700 20 run: stack=0x7f773c331700 is io blocked
2016-06-02 14:09:21.258505 7f77c6ffd700 20 dequeued request req=0x7f773c0b63b0
2016-06-02 14:09:21.258507 7f77c6ffd700 20 RGWWQ: empty
2016-06-02 14:09:21.258532 7f77c6ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2469 5.944771f8 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [setxattr inc_marker (28)] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f779c025560 con 0x5640603f1eb0
2016-06-02 14:09:21.258994 7f77477fe700 10 receive_http_header
2016-06-02 14:09:21.258997 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:21.258999 7f77477fe700 10 receive_http_header
2016-06-02 14:09:21.259000 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001e8-00575021f1-3728-us-west
2016-06-02 14:09:21.259004 7f77477fe700 10 receive_http_header
2016-06-02 14:09:21.259005 7f77477fe700 10 received header:Content-Length: 2
2016-06-02 14:09:21.259007 7f77477fe700 10 receive_http_header
2016-06-02 14:09:21.259007 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:21 GMT
2016-06-02 14:09:21.259009 7f77477fe700 10 receive_http_header
2016-06-02 14:09:21.259010 7f77477fe700 10 received header:
2016-06-02 14:09:21.297365 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:21.297505 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:21.297510 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:21.297512 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:21.297526 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c2c0f70:23RGWListBucketIndexLogCR: operate()
2016-06-02 14:09:21.297530 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:21.297532 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:21.297536 7f7760ff9700 20 run: stack=0x7f773c0a9c90 is_blocked_by_stack()=0 is_sleeping=0 waiting_for_child()=1
2016-06-02 14:09:21.297539 7f7760ff9700 20 cr:s=0x7f773c190610:op=0x7f773c619580:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:21.297581 7f7760ff9700 20 cr:s=0x7f773c190610:op=0x7f773c30f2b0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:21.297585 7f7760ff9700 20 cr:s=0x7f773c190610:op=0x7f773c30f2b0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:21.297606 7f7760ff9700 20 enqueued request req=0x7f773c3186c0
2016-06-02 14:09:21.297608 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:21.297609 7f7760ff9700 20 req: 0x7f773c3186c0
2016-06-02 14:09:21.297613 7f7760ff9700 20 run: stack=0x7f773c190610 is io blocked
2016-06-02 14:09:21.297617 7f78017fa700 20 dequeued request req=0x7f773c3186c0
2016-06-02 14:09:21.297622 7f78017fa700 20 RGWWQ: empty
2016-06-02 14:09:21.297648 7f78017fa700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2470 5.944771f8 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77e400be60 con 0x5640603f1eb0
2016-06-02 14:09:21.329063 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1139 ==== osd_op_reply(2401 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_12 [write 2097152~524288] v32'3162 uv3162 ondisk = 0) v7 ==== 208+0+0 (4020393170 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.329472 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1140 ==== osd_op_reply(2404 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_12 [write 2621440~524288] v32'3163 uv3163 ondisk = 0) v7 ==== 208+0+0 (3174954993 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.329534 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1141 ==== osd_op_reply(2405 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_12 [write 3145728~524288] v32'3164 uv3164 ondisk = 0) v7 ==== 208+0+0 (2438424655 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.329574 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1142 ==== osd_op_reply(2406 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_12 [write 3670016~524288] v32'3165 uv3165 ondisk = 0) v7 ==== 208+0+0 (3284716396 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.329943 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2471 10.a468d144 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [write 1572864~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778401d0c0 con 0x5640603e9110
2016-06-02 14:09:21.330191 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2472 10.a468d144 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [write 2097152~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778401d0c0 con 0x5640603e9110
2016-06-02 14:09:21.330460 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2473 10.a468d144 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [write 2621440~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784012ee0 con 0x5640603e9110
2016-06-02 14:09:21.330705 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2474 10.a468d144 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [write 3145728~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784130b80 con 0x5640603e9110
2016-06-02 14:09:21.349187 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1143 ==== osd_op_reply(2416 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [writefull 0~0] v32'3154 uv3154 ondisk = 0) v7 ==== 208+0+0 (1779904307 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.349247 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 835 ==== osd_op_reply(2407 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_13 [writefull 0~0] v32'3100 uv3100 ondisk = 0) v7 ==== 208+0+0 (971441813 0 0) 0x7f77fc00aad0 con 0x5640603f1eb0
2016-06-02 14:09:21.349343 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1144 ==== osd_op_reply(2417 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [writefull 0~524288] v32'3155 uv3155 ondisk = 0) v7 ==== 208+0+0 (1405433216 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.349498 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 836 ==== osd_op_reply(2408 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_13 [writefull 0~524288] v32'3101 uv3101 ondisk = 0) v7 ==== 208+0+0 (3480614 0 0) 0x7f77fc00aad0 con 0x5640603f1eb0
2016-06-02 14:09:21.349509 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1145 ==== osd_op_reply(2418 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [write 524288~524288] v32'3156 uv3156 ondisk = 0) v7 ==== 208+0+0 (3033571977 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.349624 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 837 ==== osd_op_reply(2409 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_13 [write 524288~524288] v32'3102 uv3102 ondisk = 0) v7 ==== 208+0+0 (1022030324 0 0) 0x7f77fc00aad0 con 0x5640603f1eb0
2016-06-02 14:09:21.349677 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 838 ==== osd_op_reply(2410 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_13 [write 1048576~524288] v32'3103 uv3103 ondisk = 0) v7 ==== 208+0+0 (609817220 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.349757 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2475 10.a468d144 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [write 3670016~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77848eae40 con 0x5640603e9110
2016-06-02 14:09:21.349818 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2476 10.5ac080d4 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [writefull 0~0] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784130b80 con 0x5640603e9110
2016-06-02 14:09:21.350108 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2477 10.5ac080d4 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [writefull 0~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784130b80 con 0x5640603e9110
2016-06-02 14:09:21.350363 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2478 10.5ac080d4 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 524288~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784130b80 con 0x5640603e9110
2016-06-02 14:09:21.406505 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 839 ==== osd_op_reply(2411 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_13 [write 1572864~524288] v32'3104 uv3104 ondisk = 0) v7 ==== 208+0+0 (1396760239 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.406571 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 840 ==== osd_op_reply(2412 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_13 [write 2097152~524288] v32'3105 uv3105 ondisk = 0) v7 ==== 208+0+0 (3752574329 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.406907 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2479 10.5ac080d4 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 1048576~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77848eae40 con 0x5640603e9110
2016-06-02 14:09:21.407236 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2480 10.5ac080d4 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 1572864~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77848eae40 con 0x5640603e9110
2016-06-02 14:09:21.536237 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:21.536280 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c694530:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:21.536403 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:21 2016
2016-06-02 14:09:21.536406 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:21 2016
2016-06-02 14:09:21.536421 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:21.536421 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:21 2016
/admin/log/
2016-06-02 14:09:21.536476 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:XpP+azDLgGUQIcQJ8fBz9v2cW5U=
2016-06-02 14:09:21.536495 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:21.536501 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=330, easy_handle=0x7f773c526e20
2016-06-02 14:09:21.536508 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:21.536517 7f77477fe700 20 link_request req_data=0x7f773c2d28d0 req_data->id=330, easy_handle=0x7f773c526e20
2016-06-02 14:09:21.537441 7f77477fe700 10 receive_http_header
2016-06-02 14:09:21.537446 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:21.537448 7f77477fe700 10 receive_http_header
2016-06-02 14:09:21.537448 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001e9-00575021f1-3728-us-west
2016-06-02 14:09:21.537451 7f77477fe700 10 receive_http_header
2016-06-02 14:09:21.537452 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:21.537453 7f77477fe700 10 receive_http_header
2016-06-02 14:09:21.537453 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:21 GMT
2016-06-02 14:09:21.537455 7f77477fe700 10 receive_http_header
2016-06-02 14:09:21.537456 7f77477fe700 10 received header:
2016-06-02 14:09:21.537501 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c694530:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:21.537619 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:21.537624 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869342.751510_3798.1 sync_marker.marker=1_1464869342.751510_3798.1
2016-06-02 14:09:21.537626 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869342.751510_3798.1 sync_marker.marker=1_1464869342.751510_3798.1
2016-06-02 14:09:21.537635 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:21.539181 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 841 ==== osd_op_reply(2413 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_13 [write 2621440~524288] v32'3106 uv3106 ondisk = 0) v7 ==== 208+0+0 (1646729295 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.539561 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2481 10.5ac080d4 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 2097152~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77848697d0 con 0x5640603e9110
2016-06-02 14:09:21.547976 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 842 ==== osd_op_reply(2414 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_13 [write 3145728~524288] v32'3107 uv3107 ondisk = 0) v7 ==== 208+0+0 (2056583999 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.547976 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1146 ==== osd_op_reply(2452 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [write 1048576~524288] v32'3157 uv3157 ondisk = 0) v7 ==== 208+0+0 (2892186105 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.548287 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1147 ==== osd_op_reply(2471 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [write 1572864~524288] v32'3158 uv3158 ondisk = 0) v7 ==== 208+0+0 (300934351 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.548405 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2482 10.5ac080d4 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 2621440~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77844cec10 con 0x5640603e9110
2016-06-02 14:09:21.548578 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 843 ==== osd_op_reply(2415 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_13 [write 3670016~524288] v32'3108 uv3108 ondisk = 0) v7 ==== 208+0+0 (483534034 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.548887 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1148 ==== osd_op_reply(2472 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [write 2097152~524288] v32'3159 uv3159 ondisk = 0) v7 ==== 208+0+0 (2634319641 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.548990 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2483 10.5ac080d4 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 3145728~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77844cec10 con 0x5640603e9110
2016-06-02 14:09:21.549005 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1149 ==== osd_op_reply(2473 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [write 2621440~524288] v32'3160 uv3160 ondisk = 0) v7 ==== 208+0+0 (1228594099 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.549095 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1150 ==== osd_op_reply(2474 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [write 3145728~524288] v32'3161 uv3161 ondisk = 0) v7 ==== 208+0+0 (1367947459 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.549784 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1151 ==== osd_op_reply(2475 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [write 3670016~524288] v32'3162 uv3162 ondisk = 0) v7 ==== 208+0+0 (3959791093 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.549798 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2484 10.5ac080d4 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 3670016~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77848eb630 con 0x5640603e9110
2016-06-02 14:09:21.549908 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2485 10.8db93f86 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [writefull 0~0] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77848ed150 con 0x5640603ecb10
2016-06-02 14:09:21.550156 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2486 10.8db93f86 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [writefull 0~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784012ee0 con 0x5640603ecb10
2016-06-02 14:09:21.550435 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2487 10.8db93f86 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 524288~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778465db00 con 0x5640603ecb10
2016-06-02 14:09:21.550750 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2488 10.8db93f86 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 1048576~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77841c4a70 con 0x5640603ecb10
2016-06-02 14:09:21.551206 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2489 10.8db93f86 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 1572864~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77842dd140 con 0x5640603ecb10
2016-06-02 14:09:21.551439 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2490 10.8db93f86 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 2097152~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784342f10 con 0x5640603ecb10
2016-06-02 14:09:21.551684 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2491 10.8db93f86 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 2621440~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784142ac0 con 0x5640603ecb10
2016-06-02 14:09:21.552140 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2492 10.8db93f86 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 3145728~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77847e3d30 con 0x5640603ecb10
2016-06-02 14:09:21.559254 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1152 ==== osd_op_reply(2476 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [writefull 0~0] v32'3163 uv3163 ondisk = 0) v7 ==== 208+0+0 (2856029173 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.559747 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2493 10.8db93f86 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 3670016~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784012ee0 con 0x5640603ecb10
2016-06-02 14:09:21.624213 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1153 ==== osd_op_reply(2477 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [writefull 0~524288] v32'3164 uv3164 ondisk = 0) v7 ==== 208+0+0 (2805001608 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.624481 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1154 ==== osd_op_reply(2478 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 524288~524288] v32'3165 uv3165 ondisk = 0) v7 ==== 208+0+0 (1962673743 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.624714 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2494 10.562651c7 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [writefull 0~0] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784012ee0 con 0x5640603e9110
2016-06-02 14:09:21.625059 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2495 10.562651c7 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [writefull 0~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784012b70 con 0x5640603e9110
2016-06-02 14:09:21.725856 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 618 ==== osd_op_reply(2485 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [writefull 0~0] v32'2974 uv2974 ondisk = 0) v7 ==== 208+0+0 (1397711498 0 0) 0x7f780401df00 con 0x5640603ecb10
2016-06-02 14:09:21.725908 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 619 ==== osd_op_reply(2486 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [writefull 0~524288] v32'2975 uv2975 ondisk = 0) v7 ==== 208+0+0 (1788704825 0 0) 0x7f780401bca0 con 0x5640603ecb10
2016-06-02 14:09:21.726018 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1155 ==== osd_op_reply(2479 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 1048576~524288] v32'3166 uv3166 ondisk = 0) v7 ==== 208+0+0 (2203902762 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.726040 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 620 ==== osd_op_reply(2487 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 524288~524288] v32'2976 uv2976 ondisk = 0) v7 ==== 208+0+0 (2631400694 0 0) 0x7f780401df00 con 0x5640603ecb10
2016-06-02 14:09:21.726099 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 621 ==== osd_op_reply(2488 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 1048576~524288] v32'2977 uv2977 ondisk = 0) v7 ==== 208+0+0 (2221154182 0 0) 0x7f780401bca0 con 0x5640603ecb10
2016-06-02 14:09:21.726106 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1156 ==== osd_op_reply(2480 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 1572864~524288] v32'3167 uv3167 ondisk = 0) v7 ==== 208+0+0 (3519234057 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.726214 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 622 ==== osd_op_reply(2489 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 1572864~524288] v32'2978 uv2978 ondisk = 0) v7 ==== 208+0+0 (971553456 0 0) 0x7f780401df00 con 0x5640603ecb10
2016-06-02 14:09:21.726219 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1157 ==== osd_op_reply(2481 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 2097152~524288] v32'3168 uv3168 ondisk = 0) v7 ==== 208+0+0 (2024688855 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.726305 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1158 ==== osd_op_reply(2482 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 2621440~524288] v32'3169 uv3169 ondisk = 0) v7 ==== 208+0+0 (707915764 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.726361 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1159 ==== osd_op_reply(2483 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 3145728~524288] v32'3170 uv3170 ondisk = 0) v7 ==== 208+0+0 (3717270161 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.726424 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1160 ==== osd_op_reply(2484 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [write 3670016~524288] v32'3171 uv3171 ondisk = 0) v7 ==== 208+0+0 (2400102834 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.726552 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2496 10.562651c7 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 524288~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603e9110
2016-06-02 14:09:21.726833 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2497 10.562651c7 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 1048576~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603e9110
2016-06-02 14:09:21.727105 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2498 10.562651c7 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 1572864~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603e9110
2016-06-02 14:09:21.727389 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2499 10.562651c7 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 2097152~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603e9110
2016-06-02 14:09:21.727969 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2500 10.562651c7 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 2621440~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603e9110
2016-06-02 14:09:21.728493 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2501 10.562651c7 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 3145728~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603e9110
2016-06-02 14:09:21.728911 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2502 10.562651c7 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 3670016~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784130b80 con 0x5640603e9110
2016-06-02 14:09:21.728982 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2503 10.d2750535 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [writefull 0~0] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784012b00 con 0x5640603f1eb0
2016-06-02 14:09:21.729420 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2504 10.d2750535 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [writefull 0~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77845d96a0 con 0x5640603f1eb0
2016-06-02 14:09:21.729760 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2505 10.d2750535 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 524288~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77848e5360 con 0x5640603f1eb0
2016-06-02 14:09:21.730139 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2506 10.d2750535 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 1048576~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77844cfa10 con 0x5640603f1eb0
2016-06-02 14:09:21.733372 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 623 ==== osd_op_reply(2490 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 2097152~524288] v32'2979 uv2979 ondisk = 0) v7 ==== 208+0+0 (3036928358 0 0) 0x7f780401bca0 con 0x5640603ecb10
2016-06-02 14:09:21.733533 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 624 ==== osd_op_reply(2491 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 2621440~524288] v32'2980 uv2980 ondisk = 0) v7 ==== 208+0+0 (3544516235 0 0) 0x7f780401df00 con 0x5640603ecb10
2016-06-02 14:09:21.734312 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2507 10.d2750535 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 1572864~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77844cfa80 con 0x5640603f1eb0
2016-06-02 14:09:21.737550 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2508 10.d2750535 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 2097152~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778400cd40 con 0x5640603f1eb0
2016-06-02 14:09:21.830975 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 625 ==== osd_op_reply(2492 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 3145728~524288] v32'2981 uv2981 ondisk = 0) v7 ==== 208+0+0 (3421955579 0 0) 0x7f780401bca0 con 0x5640603ecb10
2016-06-02 14:09:21.831068 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 626 ==== osd_op_reply(2493 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [write 3670016~524288] v32'2982 uv2982 ondisk = 0) v7 ==== 208+0+0 (1987723469 0 0) 0x7f780401df00 con 0x5640603ecb10
2016-06-02 14:09:21.831340 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 844 ==== osd_op_reply(2469 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [setxattr (28)] v32'2375 uv2375 ondisk = 0) v7 ==== 225+0+0 (3164054286 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.831382 7f77c6ffd700 10 cache put: name=us-east-1.rgw.log+bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 info.flags=8
2016-06-02 14:09:21.831390 7f77c6ffd700 10 moving us-east-1.rgw.log+bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 to cache LRU end
2016-06-02 14:09:21.831392 7f77c6ffd700 10 appending xattr: name=inc_marker bl.length()=28
2016-06-02 14:09:21.831410 7f77c6ffd700 10 distributing notification oid=notify.6 bl.length()=375
2016-06-02 14:09:21.831448 7f77c6ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2509 2.97c520d4 notify.6 [notify cookie 140151695255664] snapc 0=[] ondisk+read+known_if_redirected e32) v7 -- ?+0 0x7f779c027710 con 0x5640603ecb10
2016-06-02 14:09:21.831474 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1161 ==== osd_op_reply(2494 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [writefull 0~0] v32'3166 uv3166 ondisk = 0) v7 ==== 208+0+0 (2399066083 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.831636 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1162 ==== osd_op_reply(2495 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [writefull 0~524288] v32'3167 uv3167 ondisk = 0) v7 ==== 208+0+0 (3073166672 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.831885 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 627 ==== watch-notify(notify (1) cookie 94834492718368 notify 137438953606 ret 0) v3 ==== 417+0+0 (4179246394 0 0) 0x7f780401df00 con 0x5640603ecb10
2016-06-02 14:09:21.831908 7f78297fa700 10 RGWWatcher::handle_notify()  notify_id 137438953606 cookie 94834492718368 notifier 24110 bl.length()=375
2016-06-02 14:09:21.831929 7f78297fa700 10 cache put: name=us-east-1.rgw.log+bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 info.flags=8
2016-06-02 14:09:21.831929 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 628 ==== osd_op_reply(2509 notify.6 [notify cookie 140151695255664] v0'0 uv1 ondisk = 0) v7 ==== 128+0+8 (386979837 0 1346501324) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:21.831935 7f78297fa700 10 moving us-east-1.rgw.log+bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 to cache LRU end
2016-06-02 14:09:21.831938 7f78297fa700 10 appending xattr: name=inc_marker bl.length()=28
2016-06-02 14:09:21.831959 7f78297fa700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2510 2.97c520d4 notify.6 [notify-ack 0~0] snapc 0=[] read+known_if_redirected e32) v7 -- ?+0 0x7f77e0003790 con 0x5640603ecb10
2016-06-02 14:09:21.832353 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 629 ==== watch-notify(notify_complete (2) cookie 140151695255664 notify 137438953606 ret 0) v3 ==== 42+0+28 (2211652883 0 174158344) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:21.832400 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 630 ==== osd_op_reply(2510 notify.6 [notify-ack 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3357944540 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:21.832405 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c028a90:26RGWSimpleRadosWriteAttrsCR: operate()
2016-06-02 14:09:21.832412 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c028a90:26RGWSimpleRadosWriteAttrsCR: operate()
2016-06-02 14:09:21.832414 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c028a90:26RGWSimpleRadosWriteAttrsCR: operate()
2016-06-02 14:09:21.832416 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c028a90:26RGWSimpleRadosWriteAttrsCR: operate()
2016-06-02 14:09:21.832438 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c2ff160:26RGWBucketSyncSingleEntryCRINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE11rgw_obj_keyE: operate()
2016-06-02 14:09:21.832445 7f7760ff9700 20 run: stack=0x7f773c331700 is done
2016-06-02 14:09:21.832451 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:21.832454 7f7760ff9700 20 collect(): s=0x7f773c0a9c90 stack=0x7f773c190610 is still running
2016-06-02 14:09:21.832455 7f7760ff9700 20 collect(): s=0x7f773c0a9c90 stack=0x7f773c331700 is complete
2016-06-02 14:09:21.832457 7f7760ff9700 20 run: stack=0x7f773c0a9c90 is_blocked_by_stack()=0 is_sleeping=0 waiting_for_child()=1
2016-06-02 14:09:21.832584 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2511 10.d2750535 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 2621440~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77847e3640 con 0x5640603f1eb0
2016-06-02 14:09:21.832799 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2512 10.d2750535 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 3145728~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77847e3640 con 0x5640603f1eb0
2016-06-02 14:09:21.832990 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2513 10.d2750535 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 3670016~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77841c0b90 con 0x5640603f1eb0
2016-06-02 14:09:21.833028 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2514 10.ea2c6d96 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [writefull 0~0] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77845519b0 con 0x5640603ecb10
2016-06-02 14:09:21.930705 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1163 ==== osd_op_reply(2496 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 524288~524288] v32'3168 uv3168 ack = 0) v7 ==== 208+0+0 (1387195093 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.930705 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 845 ==== osd_op_reply(2503 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [writefull 0~0] v32'3124 uv3124 ondisk = 0) v7 ==== 208+0+0 (3867054056 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.930752 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1164 ==== osd_op_reply(2496 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 524288~524288] v32'3168 uv3168 ondisk = 0) v7 ==== 208+0+0 (1097222559 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.930797 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 846 ==== osd_op_reply(2504 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [writefull 0~524288] v32'3125 uv3125 ondisk = 0) v7 ==== 208+0+0 (3752629595 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.930825 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1165 ==== osd_op_reply(2497 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 1048576~524288] v32'3169 uv3169 ondisk = 0) v7 ==== 208+0+0 (1507175151 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.930883 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 847 ==== osd_op_reply(2505 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 524288~524288] v32'3126 uv3126 ondisk = 0) v7 ==== 208+0+0 (3815995529 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.930950 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 848 ==== osd_op_reply(2506 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 1048576~524288] v32'3127 uv3127 ondisk = 0) v7 ==== 208+0+0 (4223704057 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.930978 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1166 ==== osd_op_reply(2498 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 1572864~524288] v32'3170 uv3170 ondisk = 0) v7 ==== 208+0+0 (3831042009 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.931061 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1167 ==== osd_op_reply(2499 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 2097152~524288] v32'3171 uv3171 ondisk = 0) v7 ==== 208+0+0 (1756523535 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.931119 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1168 ==== osd_op_reply(2500 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 2621440~524288] v32'3172 uv3172 ondisk = 0) v7 ==== 208+0+0 (250917858 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.931173 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1169 ==== osd_op_reply(2501 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 3145728~524288] v32'3173 uv3173 ondisk = 0) v7 ==== 208+0+0 (373772434 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.931213 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 849 ==== osd_op_reply(2507 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 1572864~524288] v32'3128 uv3128 ondisk = 0) v7 ==== 208+0+0 (805226323 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.931229 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1170 ==== osd_op_reply(2502 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [write 3670016~524288] v32'3174 uv3174 ondisk = 0) v7 ==== 208+0+0 (2882270628 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:21.931277 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 850 ==== osd_op_reply(2508 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 2097152~524288] v32'3129 uv3129 ondisk = 0) v7 ==== 208+0+0 (2736104581 0 0) 0x7f77fc009360 con 0x5640603f1eb0
2016-06-02 14:09:21.931468 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2515 10.ea2c6d96 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [writefull 0~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603ecb10
2016-06-02 14:09:21.931718 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2516 10.ea2c6d96 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 524288~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603ecb10
2016-06-02 14:09:21.931975 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2517 10.ea2c6d96 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 1048576~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603ecb10
2016-06-02 14:09:21.932180 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2518 10.ea2c6d96 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 1572864~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603ecb10
2016-06-02 14:09:21.932432 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2519 10.ea2c6d96 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 2097152~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77842c9390 con 0x5640603ecb10
2016-06-02 14:09:21.933150 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2520 10.ea2c6d96 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 2621440~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603ecb10
2016-06-02 14:09:21.933542 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2521 10.ea2c6d96 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 3145728~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77842c9390 con 0x5640603ecb10
2016-06-02 14:09:21.933894 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2522 10.ea2c6d96 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 3670016~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778434d1c0 con 0x5640603ecb10
2016-06-02 14:09:21.933953 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2523 10.4990a3e5 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [writefull 0~0] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778423e1f0 con 0x5640603f1eb0
2016-06-02 14:09:21.934675 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2524 10.4990a3e5 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [writefull 0~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77842cab60 con 0x5640603f1eb0
2016-06-02 14:09:21.934946 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2525 10.4990a3e5 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 524288~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77844cf930 con 0x5640603f1eb0
2016-06-02 14:09:21.935157 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2526 10.4990a3e5 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 1048576~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77842dd3a0 con 0x5640603f1eb0
2016-06-02 14:09:21.935558 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2527 10.4990a3e5 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 1572864~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784011140 con 0x5640603f1eb0
2016-06-02 14:09:22.018788 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 851 ==== osd_op_reply(2511 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 2621440~524288] v32'3130 uv3130 ondisk = 0) v7 ==== 208+0+0 (513343923 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.018828 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 631 ==== osd_op_reply(2514 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [writefull 0~0] v32'2983 uv2983 ondisk = 0) v7 ==== 208+0+0 (1573406341 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.018913 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 852 ==== osd_op_reply(2512 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 3145728~524288] v32'3131 uv3131 ondisk = 0) v7 ==== 208+0+0 (103473859 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.019045 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 853 ==== osd_op_reply(2513 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [write 3670016~524288] v32'3132 uv3132 ondisk = 0) v7 ==== 208+0+0 (1617706286 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.019252 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2528 10.4990a3e5 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 2097152~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778402ebe0 con 0x5640603f1eb0
2016-06-02 14:09:22.019527 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2529 10.4990a3e5 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 2621440~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778402ebe0 con 0x5640603f1eb0
2016-06-02 14:09:22.019778 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2530 10.4990a3e5 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 3145728~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77847e4020 con 0x5640603f1eb0
2016-06-02 14:09:22.020054 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2531 10.4990a3e5 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 3670016~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778401a540 con 0x5640603f1eb0
2016-06-02 14:09:22.115095 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 854 ==== osd_op_reply(2523 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [writefull 0~0] v32'3133 uv3133 ondisk = 0) v7 ==== 208+0+0 (2549554877 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.115163 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 855 ==== osd_op_reply(2524 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [writefull 0~524288] v32'3134 uv3134 ondisk = 0) v7 ==== 208+0+0 (1094058523 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.115222 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 856 ==== osd_op_reply(2525 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 524288~524288] v32'3135 uv3135 ondisk = 0) v7 ==== 208+0+0 (2465917404 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.115313 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 857 ==== osd_op_reply(2526 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 1048576~524288] v32'3136 uv3136 ondisk = 0) v7 ==== 208+0+0 (777441961 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.115494 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 858 ==== osd_op_reply(2470 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call] v32'2376 uv2376 ondisk = 0) v7 ==== 225+0+0 (3177270013 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.115553 7f7760ff9700 20 cr:s=0x7f773c190610:op=0x7f773c30f2b0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:22.115568 7f7760ff9700 20 cr:s=0x7f773c190610:op=0x7f773c30f2b0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:22.115561 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 632 ==== osd_op_reply(2515 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [writefull 0~524288] v32'2984 uv2984 ondisk = 0) v7 ==== 208+0+0 (3803947967 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.115571 7f7760ff9700 20 cr:s=0x7f773c190610:op=0x7f773c30f2b0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:22.115572 7f7760ff9700 20 cr:s=0x7f773c190610:op=0x7f773c30f2b0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:22.115588 7f7760ff9700 20 cr:s=0x7f773c190610:op=0x7f773c619580:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:22.115590 7f7760ff9700 20 run: stack=0x7f773c190610 is done
2016-06-02 14:09:22.115594 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:22.115597 7f7760ff9700 20 collect(): s=0x7f773c0a9c90 stack=0x7f773c190610 is complete
2016-06-02 14:09:22.115599 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:22.115609 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c670d30:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:22.115617 7f7760ff9700 20 cr:s=0x7f773c0a9c90:op=0x7f773c2e2180:24RGWDataSyncSingleEntryCR: operate()
2016-06-02 14:09:22.115622 7f7760ff9700 20 run: stack=0x7f773c0a9c90 is done
2016-06-02 14:09:22.115631 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 633 ==== osd_op_reply(2516 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 524288~524288] v32'2985 uv2985 ondisk = 0) v7 ==== 208+0+0 (829900920 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.115680 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 634 ==== osd_op_reply(2517 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 1048576~524288] v32'2986 uv2986 ondisk = 0) v7 ==== 208+0+0 (3336023325 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.115729 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 635 ==== osd_op_reply(2518 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 1572864~524288] v32'2987 uv2987 ondisk = 0) v7 ==== 208+0+0 (2487750206 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.115775 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 636 ==== osd_op_reply(2519 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 2097152~524288] v32'2988 uv2988 ondisk = 0) v7 ==== 208+0+0 (746197286 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.115776 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 859 ==== osd_op_reply(2527 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 1572864~524288] v32'3137 uv3137 ondisk = 0) v7 ==== 208+0+0 (2093576586 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.115862 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 637 ==== osd_op_reply(2520 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 2621440~524288] v32'2989 uv2989 ondisk = 0) v7 ==== 208+0+0 (2128982533 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.115948 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 638 ==== osd_op_reply(2521 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 3145728~524288] v32'2990 uv2990 ondisk = 0) v7 ==== 208+0+0 (2303022944 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.115959 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2532 10.cc584f41 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [writefull 0~0] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77842dc1d0 con 0x5640603e9110
2016-06-02 14:09:22.116013 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 639 ==== osd_op_reply(2522 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [write 3670016~524288] v32'2991 uv2991 ondisk = 0) v7 ==== 208+0+0 (3688559683 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.116467 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2533 10.cc584f41 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [writefull 0~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603e9110
2016-06-02 14:09:22.116825 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2534 10.cc584f41 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 524288~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603e9110
2016-06-02 14:09:22.117206 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2535 10.cc584f41 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 1048576~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603e9110
2016-06-02 14:09:22.117552 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2536 10.cc584f41 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 1572864~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77841c2100 con 0x5640603e9110
2016-06-02 14:09:22.117826 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2537 10.cc584f41 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 2097152~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784130b80 con 0x5640603e9110
2016-06-02 14:09:22.118760 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2538 10.cc584f41 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 2621440~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784763140 con 0x5640603e9110
2016-06-02 14:09:22.119310 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2539 10.cc584f41 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 3145728~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784012ee0 con 0x5640603e9110
2016-06-02 14:09:22.119676 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2540 10.cc584f41 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 3670016~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784868410 con 0x5640603e9110
2016-06-02 14:09:22.119727 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2541 10.2109ab6d 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [writefull 0~0] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77846dab80 con 0x5640603f1eb0
2016-06-02 14:09:22.120111 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2542 10.2109ab6d 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [writefull 0~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77844cf850 con 0x5640603f1eb0
2016-06-02 14:09:22.120478 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2543 10.2109ab6d 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 524288~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77848eb820 con 0x5640603f1eb0
2016-06-02 14:09:22.120768 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2544 10.2109ab6d 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 1048576~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840d2570 con 0x5640603f1eb0
2016-06-02 14:09:22.177848 7f7704fd9700 20 execute(): read data: [{"key":36,"val":["lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1"]}]
2016-06-02 14:09:22.177920 7f7704fd9700 20 execute(): updated shard=36
2016-06-02 14:09:22.177922 7f7704fd9700 20 execute(): modified key=lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1
2016-06-02 14:09:22.177923 7f7704fd9700 20 wakeup_data_sync_shards: source_zone=91441c60-0ca1-4118-9685-990e04272142, shard_ids={36=lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1}
2016-06-02 14:09:22.177944 7f7704fd9700  2 req 574:1.000855::POST /admin/log:datalog_notify:completing
2016-06-02 14:09:22.177948 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:22.177954 7f7760ff9700 20 incremental_sync(): async update notification: lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1
2016-06-02 14:09:22.178002 7f7704fd9700  2 req 574:1.000902::POST /admin/log:datalog_notify:op status=0
2016-06-02 14:09:22.178005 7f7704fd9700  2 req 574:1.000916::POST /admin/log:datalog_notify:http status=200
2016-06-02 14:09:22.178008 7f7704fd9700  1 ====== req done req=0x7f7704fd3790 op status=0 http_status=200 ======
2016-06-02 14:09:22.178045 7f7704fd9700  1 civetweb: 0x7f77dc0024a0: 127.0.0.1 - - [02/Jun/2016:14:09:21 +0200] "POST /admin/log HTTP/1.1" 200 0 - -
2016-06-02 14:09:22.178067 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c2e2180:24RGWDataSyncSingleEntryCR: operate()
2016-06-02 14:09:22.178079 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:22.178093 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c670d30:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:22.178099 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c694530:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:22.178164 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:22 2016
2016-06-02 14:09:22.178167 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:22 2016
2016-06-02 14:09:22.178171 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:22.178172 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:22 2016
/admin/log/
2016-06-02 14:09:22.178221 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:zp1/zpfu1vCEJ7ynaf6OAGpu+1Y=
2016-06-02 14:09:22.178232 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:22.178240 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=331, easy_handle=0x7f773c526e20
2016-06-02 14:09:22.178248 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:22.178252 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:32RGWReadBucketSyncStatusCoroutine: operate()
2016-06-02 14:09:22.178260 7f77477fe700 20 link_request req_data=0x7f773c175430 req_data->id=331, easy_handle=0x7f773c526e20
2016-06-02 14:09:22.178262 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0c79f0:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:22.178270 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0c79f0:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:22.178287 7f7760ff9700 20 enqueued request req=0x7f773c2ec170
2016-06-02 14:09:22.178290 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:22.178291 7f7760ff9700 20 req: 0x7f773c2ec170
2016-06-02 14:09:22.178295 7f7760ff9700 20 run: stack=0x7f773c331700 is io blocked
2016-06-02 14:09:22.178301 7f77b27fc700 20 dequeued request req=0x7f773c2ec170
2016-06-02 14:09:22.178302 7f77b27fc700 20 RGWWQ: empty
2016-06-02 14:09:22.178315 7f77b27fc700 10 cache get: name=us-east-1.rgw.log+bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=3, cached=11)
2016-06-02 14:09:22.178344 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0c79f0:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:22.178347 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0c79f0:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:22.178349 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0c79f0:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:22.178351 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0c79f0:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:22.178366 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:32RGWReadBucketSyncStatusCoroutine: operate()
2016-06-02 14:09:22.178375 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c670d30:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:22.178378 7f7760ff9700 20 operate(): sync status for bucket lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1: 2
2016-06-02 14:09:22.178385 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:22.178387 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:22.178394 7f7760ff9700 20 enqueued request req=0x7f773c27dde0
2016-06-02 14:09:22.178395 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:22.178396 7f7760ff9700 20 req: 0x7f773c27dde0
2016-06-02 14:09:22.178399 7f7760ff9700 20 run: stack=0x7f773c331700 is io blocked
2016-06-02 14:09:22.178413 7f77aaffd700 20 dequeued request req=0x7f773c27dde0
2016-06-02 14:09:22.178416 7f77aaffd700 20 RGWWQ: empty
2016-06-02 14:09:22.178420 7f77aaffd700 20 reading from us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:22.178439 7f77aaffd700 20 get_system_obj_state: rctx=0x7f77aaffb560 obj=us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 state=0x7f776c02f538 s->prefetch_data=0
2016-06-02 14:09:22.178446 7f77aaffd700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=22, cached=23)
2016-06-02 14:09:22.178453 7f77aaffd700 20 get_system_obj_state: s->obj_tag was set empty
2016-06-02 14:09:22.178456 7f77aaffd700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=17, cached=23)
2016-06-02 14:09:22.178477 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:22.178481 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:22.178483 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:22.178484 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:22.178490 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c670d30:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:22.178492 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c670d30:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:22.178493 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c670d30:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:22.178504 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c3077c0:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:22.178550 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c619580:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:22.178584 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c3077c0:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:22.178587 7f7760ff9700 20 run: stack=0x7f773c331700 is_blocked_by_stack()=0 is_sleeping=1 waiting_for_child()=0
2016-06-02 14:09:22.178590 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c30f2b0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:22.178592 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c30f2b0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:22.178601 7f7760ff9700 20 enqueued request req=0x7f773c3a6e10
2016-06-02 14:09:22.178602 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:22.178603 7f7760ff9700 20 req: 0x7f773c3a6e10
2016-06-02 14:09:22.178606 7f7760ff9700 20 run: stack=0x7f773c0aa500 is io blocked
2016-06-02 14:09:22.178614 7f77c5ffb700 20 dequeued request req=0x7f773c3a6e10
2016-06-02 14:09:22.178620 7f77c5ffb700 20 RGWWQ: empty
2016-06-02 14:09:22.178661 7f77c5ffb700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2545 5.944771f8 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77b8020a70 con 0x5640603f1eb0
2016-06-02 14:09:22.179180 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.179183 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:22.179185 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.179186 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001ea-00575021f2-3728-us-west
2016-06-02 14:09:22.179189 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.179189 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:22.179190 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.179191 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:22 GMT
2016-06-02 14:09:22.179192 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.179193 7f77477fe700 10 received header:
2016-06-02 14:09:22.179234 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c694530:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:22.179346 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:22.179350 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869342.751510_3798.1 sync_marker.marker=1_1464869342.751510_3798.1
2016-06-02 14:09:22.179351 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869342.751510_3798.1 sync_marker.marker=1_1464869342.751510_3798.1
2016-06-02 14:09:22.179356 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:22.273106 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 860 ==== osd_op_reply(2528 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 2097152~524288] v32'3138 uv3138 ondisk = 0) v7 ==== 208+0+0 (523310153 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.273497 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 861 ==== osd_op_reply(2529 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 2621440~524288] v32'3139 uv3139 ondisk = 0) v7 ==== 208+0+0 (1303295850 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.273576 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2546 10.2109ab6d 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 1572864~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77841c4a70 con 0x5640603f1eb0
2016-06-02 14:09:22.273909 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2547 10.2109ab6d 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 2097152~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77841c4a70 con 0x5640603f1eb0
2016-06-02 14:09:22.274050 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 862 ==== osd_op_reply(2530 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 3145728~524288] v32'3140 uv3140 ondisk = 0) v7 ==== 208+0+0 (1640245460 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.274376 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2548 10.2109ab6d 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 2621440~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77841c4a70 con 0x5640603f1eb0
2016-06-02 14:09:22.274603 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1171 ==== osd_op_reply(2532 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [writefull 0~0] v32'2914 uv2914 ondisk = 0) v7 ==== 208+0+0 (1841646966 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:22.274653 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1172 ==== osd_op_reply(2533 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [writefull 0~524288] v32'2915 uv2915 ondisk = 0) v7 ==== 208+0+0 (1410815941 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:22.274800 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1173 ==== osd_op_reply(2534 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 524288~524288] v32'2916 uv2916 ondisk = 0) v7 ==== 208+0+0 (3003302092 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:22.275002 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1174 ==== osd_op_reply(2535 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 1048576~524288] v32'2917 uv2917 ondisk = 0) v7 ==== 208+0+0 (2880512956 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:22.275055 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1175 ==== osd_op_reply(2536 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 1572864~524288] v32'2918 uv2918 ondisk = 0) v7 ==== 208+0+0 (373130890 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:22.275152 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 863 ==== osd_op_reply(2531 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [write 3670016~524288] v32'3141 uv3141 ondisk = 0) v7 ==== 208+0+0 (861637623 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.275218 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 864 ==== osd_op_reply(2541 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [writefull 0~0] v32'3142 uv3142 ondisk = 0) v7 ==== 208+0+0 (499789800 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.275671 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2549 10.2109ab6d 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 3145728~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77841c4a70 con 0x5640603f1eb0
2016-06-02 14:09:22.275931 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2550 10.2109ab6d 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 3670016~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77841c4a70 con 0x5640603f1eb0
2016-06-02 14:09:22.275976 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2551 10.7e0ff622 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [writefull 0~0] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778465d230 con 0x5640603ecb10
2016-06-02 14:09:22.276612 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2552 10.7e0ff622 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [writefull 0~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778465d230 con 0x5640603ecb10
2016-06-02 14:09:22.276881 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2553 10.7e0ff622 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 524288~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77841c4a70 con 0x5640603ecb10
2016-06-02 14:09:22.277178 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2554 10.7e0ff622 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 1048576~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77847e4160 con 0x5640603ecb10
2016-06-02 14:09:22.329849 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 865 ==== osd_op_reply(2542 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [writefull 0~524288] v32'3143 uv3143 ondisk = 0) v7 ==== 208+0+0 (605566299 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.450065 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1176 ==== osd_op_reply(2537 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 2097152~524288] v32'2919 uv2919 ondisk = 0) v7 ==== 208+0+0 (2597774684 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:22.450103 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 866 ==== osd_op_reply(2543 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 524288~524288] v32'3144 uv3144 ondisk = 0) v7 ==== 208+0+0 (1903501589 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.450294 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 867 ==== osd_op_reply(2544 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 1048576~524288] v32'3145 uv3145 ondisk = 0) v7 ==== 208+0+0 (1774633573 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.450497 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1177 ==== osd_op_reply(2538 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 2621440~524288] v32'2920 uv2920 ondisk = 0) v7 ==== 208+0+0 (1323874806 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:22.450533 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 868 ==== osd_op_reply(2546 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 1572864~524288] v32'3146 uv3146 ondisk = 0) v7 ==== 208+0+0 (3561645907 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.450545 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1178 ==== osd_op_reply(2539 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 3145728~524288] v32'2921 uv2921 ondisk = 0) v7 ==== 208+0+0 (1448827526 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:22.450637 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1179 ==== osd_op_reply(2540 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [write 3670016~524288] v32'2922 uv2922 ondisk = 0) v7 ==== 208+0+0 (3956801456 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:22.450793 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2555 10.7e0ff622 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 1572864~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840132b0 con 0x5640603ecb10
2016-06-02 14:09:22.450977 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2556 10.7e0ff622 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 2097152~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784763140 con 0x5640603ecb10
2016-06-02 14:09:22.451177 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2557 10.7e0ff622 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 2621440~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778476ce60 con 0x5640603ecb10
2016-06-02 14:09:22.451382 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2558 10.7e0ff622 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 3145728~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840d2160 con 0x5640603ecb10
2016-06-02 14:09:22.451610 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2559 10.7e0ff622 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 3670016~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77846dab80 con 0x5640603ecb10
2016-06-02 14:09:22.451661 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2560 10.ab707ff2 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 [writefull 0~0] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77840d1cd0 con 0x5640603ecb10
2016-06-02 14:09:22.451868 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2561 10.ab707ff2 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 [writefull 0~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778403af50 con 0x5640603ecb10
2016-06-02 14:09:22.452073 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2562 10.ab707ff2 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 [write 524288~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77848e5710 con 0x5640603ecb10
2016-06-02 14:09:22.452325 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2563 10.ab707ff2 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 [write 1048576~524288] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f778400f8e0 con 0x5640603ecb10
2016-06-02 14:09:22.486551 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 640 ==== osd_op_reply(2551 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [writefull 0~0] v32'3162 uv3162 ondisk = 0) v7 ==== 208+0+0 (3892527666 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.486619 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 641 ==== osd_op_reply(2552 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [writefull 0~524288] v32'3163 uv3163 ondisk = 0) v7 ==== 208+0+0 (3520157825 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.522707 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 642 ==== osd_op_reply(2553 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 524288~524288] v32'3164 uv3164 ondisk = 0) v7 ==== 208+0+0 (918867848 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.522791 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 643 ==== osd_op_reply(2554 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 1048576~524288] v32'3165 uv3165 ondisk = 0) v7 ==== 208+0+0 (779562232 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.522893 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 869 ==== osd_op_reply(2547 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 2097152~524288] v32'3147 uv3147 ondisk = 0) v7 ==== 208+0+0 (1486947461 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.523109 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 870 ==== osd_op_reply(2548 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 2621440~524288] v32'3148 uv3148 ondisk = 0) v7 ==== 208+0+0 (1055382376 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.523172 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 871 ==== osd_op_reply(2549 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 3145728~524288] v32'3149 uv3149 ondisk = 0) v7 ==== 208+0+0 (643053592 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.523231 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 872 ==== osd_op_reply(2550 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [write 3670016~524288] v32'3150 uv3150 ondisk = 0) v7 ==== 208+0+0 (2614664494 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.523762 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2564 10.ab707ff2 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 [write 1572864~312512] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784015470 con 0x5640603ecb10
2016-06-02 14:09:22.539053 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:22.539139 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c694530:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:22.539211 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:22 2016
2016-06-02 14:09:22.539214 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:22 2016
2016-06-02 14:09:22.539218 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:22.539218 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:22 2016
/admin/log/
2016-06-02 14:09:22.539247 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:zp1/zpfu1vCEJ7ynaf6OAGpu+1Y=
2016-06-02 14:09:22.539256 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:22.539262 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=332, easy_handle=0x7f773c380450
2016-06-02 14:09:22.539270 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:22.539278 7f77477fe700 20 link_request req_data=0x7f773c2ecf10 req_data->id=332, easy_handle=0x7f773c380450
2016-06-02 14:09:22.540085 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.540089 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:22.540091 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.540092 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001eb-00575021f2-3728-us-west
2016-06-02 14:09:22.540096 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.540096 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:22.540097 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.540098 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:22 GMT
2016-06-02 14:09:22.540099 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.540100 7f77477fe700 10 received header:
2016-06-02 14:09:22.540154 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c694530:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:22.540454 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:22.540458 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869342.751510_3798.1 sync_marker.marker=1_1464869342.751510_3798.1
2016-06-02 14:09:22.540462 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869342.751510_3798.1 sync_marker.marker=1_1464869342.751510_3798.1
2016-06-02 14:09:22.540478 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:22.605580 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 644 ==== osd_op_reply(2555 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 1572864~524288] v32'3166 uv3166 ondisk = 0) v7 ==== 208+0+0 (2482735566 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.605786 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 645 ==== osd_op_reply(2556 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 2097152~524288] v32'3167 uv3167 ondisk = 0) v7 ==== 208+0+0 (521202200 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.605837 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 646 ==== osd_op_reply(2557 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 2621440~524288] v32'3168 uv3168 ondisk = 0) v7 ==== 208+0+0 (1745430067 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.605882 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 647 ==== osd_op_reply(2558 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 3145728~524288] v32'3169 uv3169 ondisk = 0) v7 ==== 208+0+0 (1891288387 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.606216 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 648 ==== osd_op_reply(2559 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [write 3670016~524288] v32'3170 uv3170 ondisk = 0) v7 ==== 208+0+0 (3442892917 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.606271 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 649 ==== osd_op_reply(2560 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 [writefull 0~0] v32'3171 uv3171 ondisk = 0) v7 ==== 208+0+0 (2710505235 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.606323 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 650 ==== osd_op_reply(2561 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 [writefull 0~524288] v32'3172 uv3172 ondisk = 0) v7 ==== 208+0+0 (2894356846 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.606491 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 651 ==== osd_op_reply(2562 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 [write 524288~524288] v32'3173 uv3173 ondisk = 0) v7 ==== 208+0+0 (2135460521 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.606537 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 652 ==== osd_op_reply(2563 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 [write 1048576~524288] v32'3174 uv3174 ondisk = 0) v7 ==== 208+0+0 (2296934348 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.779928 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 653 ==== osd_op_reply(2564 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 [write 1572864~312512] v32'3175 uv3175 ondisk = 0) v7 ==== 208+0+0 (32206817 0 0) 0x7f7804019be0 con 0x5640603ecb10
2016-06-02 14:09:22.780047 7f77b2ffd700 20 get_obj_state: rctx=0x7f77b2ffb190 obj=lotr:atroll5 state=0x7f778400dfc8 s->prefetch_data=0
2016-06-02 14:09:22.780082 7f77b2ffd700 20 get_obj_state: rctx=0x7f77b2ffb190 obj=lotr:atroll5 state=0x7f778400dfc8 s->prefetch_data=0
2016-06-02 14:09:22.780090 7f77b2ffd700 10 setting object write_tag=_cgbXHc6NT5tBwQ1P6tfr8uP8lxG63Dt
2016-06-02 14:09:22.780155 7f77b2ffd700 20 reading from us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:22.780170 7f77b2ffd700 20 get_system_obj_state: rctx=0x7f77b2ff7710 obj=us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 state=0x7f77846e00b8 s->prefetch_data=0
2016-06-02 14:09:22.780181 7f77b2ffd700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=22, cached=23)
2016-06-02 14:09:22.780192 7f77b2ffd700 20 get_system_obj_state: s->obj_tag was set empty
2016-06-02 14:09:22.780197 7f77b2ffd700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=17, cached=23)
2016-06-02 14:09:22.780217 7f77b2ffd700 20  bucket index object: .dir.62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:22.780246 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2565 9.9063c57f .dir.62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call rgw.bucket_prepare_op] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784003830 con 0x5640603f1eb0
2016-06-02 14:09:22.787965 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 873 ==== osd_op_reply(2545 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call] v32'2377 uv2377 ondisk = 0) v7 ==== 225+0+0 (3717758556 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.788028 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c30f2b0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:22.788039 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c30f2b0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:22.788041 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c30f2b0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:22.788043 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c30f2b0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:22.788059 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c619580:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:22.788069 7f7760ff9700 20 run: stack=0x7f773c0aa500 is io blocked
2016-06-02 14:09:22.788072 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c3077c0:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:22.788075 7f7760ff9700 20 abhi: starting a new RGWBucketIncSyncShardMarkerTrack with marker oidbucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:22.788076 7f7760ff9700 20 operate(): listing bilog for incremental sync
2016-06-02 14:09:22.788154 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c2c0f70:23RGWListBucketIndexLogCR: operate()
2016-06-02 14:09:22.788166 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:22.788170 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:22.788267 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:22 2016
2016-06-02 14:09:22.788273 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:22 2016
2016-06-02 14:09:22.788280 7f7760ff9700 10 get_canon_resource(): dest=/admin/log
2016-06-02 14:09:22.788282 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:22 2016
/admin/log
2016-06-02 14:09:22.788318 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:yIc9VEXythM2obxbfYOf3hAeDNg=
2016-06-02 14:09:22.788330 7f7760ff9700 20 sending request to http://localhost:8002/admin/log?bucket-instance=lotr%3A62bc922d-f295-4067-ae36-e23e2f231aad.24099.1&format=json&marker=00000000472.1089.3&type=bucket-index&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:22.788340 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=333, easy_handle=0x7f773c380450
2016-06-02 14:09:22.788350 7f7760ff9700 20 run: stack=0x7f773c331700 is io blocked
2016-06-02 14:09:22.788364 7f77477fe700 20 link_request req_data=0x7f773c080980 req_data->id=333, easy_handle=0x7f773c380450
2016-06-02 14:09:22.789858 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.789867 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:22.789870 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.789870 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001ec-00575021f2-3728-us-west
2016-06-02 14:09:22.789878 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.789878 7f77477fe700 10 received header:Content-Length: 2
2016-06-02 14:09:22.789881 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.789882 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:22 GMT
2016-06-02 14:09:22.789884 7f77477fe700 10 receive_http_header
2016-06-02 14:09:22.789884 7f77477fe700 10 received header:
2016-06-02 14:09:22.789962 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:22.790460 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:22.790468 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:22.790471 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:22.790478 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c2c0f70:23RGWListBucketIndexLogCR: operate()
2016-06-02 14:09:22.790483 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c3077c0:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:22.790486 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c3077c0:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:22.790490 7f7760ff9700 20 run: stack=0x7f773c331700 is_blocked_by_stack()=0 is_sleeping=0 waiting_for_child()=1
2016-06-02 14:09:22.790497 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c619580:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:22.790577 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c30f2b0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:22.790580 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c30f2b0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:22.790598 7f7760ff9700 20 enqueued request req=0x7f773c016e50
2016-06-02 14:09:22.790600 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:22.790601 7f7760ff9700 20 req: 0x7f773c016e50
2016-06-02 14:09:22.790605 7f7760ff9700 20 run: stack=0x7f773c0aa500 is io blocked
2016-06-02 14:09:22.790611 7f77c4ff9700 20 dequeued request req=0x7f773c016e50
2016-06-02 14:09:22.790615 7f77c4ff9700 20 RGWWQ: empty
2016-06-02 14:09:22.790667 7f77c4ff9700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2566 5.944771f8 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7794001e20 con 0x5640603f1eb0
2016-06-02 14:09:22.853190 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 874 ==== osd_op_reply(2565 .dir.62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call] v32'47 uv47 ondisk = 0) v7 ==== 169+0+0 (480437509 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.853301 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2567 10.a8f56d53 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1_atroll5 [create 0~0 [excl],setxattr user.rgw.idtag (33),writefull 0~524288,setxattr user.rgw.manifest (573),setxattr user.rgw.acl (115),setxattr user.rgw.content_type (25),setxattr user.rgw.etag (36),setxattr user.rgw.idtag (47),setxattr user.rgw.pg_ver (8),setxattr user.rgw.source_zone (4)] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7784015470 con 0x5640603f1eb0
2016-06-02 14:09:22.874714 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 875 ==== osd_op_reply(2566 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call] v32'2378 uv2378 ondisk = 0) v7 ==== 225+0+0 (1383222504 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.874784 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c30f2b0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:22.874796 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c30f2b0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:22.874798 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c30f2b0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:22.874799 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c30f2b0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:22.874813 7f7760ff9700 20 cr:s=0x7f773c0aa500:op=0x7f773c619580:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:22.874815 7f7760ff9700 20 run: stack=0x7f773c0aa500 is done
2016-06-02 14:09:22.874819 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c3077c0:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:22.874821 7f7760ff9700 20 collect(): s=0x7f773c331700 stack=0x7f773c0aa500 is complete
2016-06-02 14:09:22.874823 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c3077c0:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:22.874830 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c670d30:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:22.874836 7f7760ff9700 20 cr:s=0x7f773c331700:op=0x7f773c2e2180:24RGWDataSyncSingleEntryCR: operate()
2016-06-02 14:09:22.874840 7f7760ff9700 20 run: stack=0x7f773c331700 is done
2016-06-02 14:09:22.937484 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 876 ==== osd_op_reply(2567 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1_atroll5 [create 0~0 [excl],setxattr (33),writefull 0~524288,setxattr (573),setxattr (115),setxattr (25),setxattr (36),setxattr (47),setxattr (8),setxattr (4)] v32'3109 uv3109 ondisk = 0) v7 ==== 550+0+0 (87994013 0 0) 0x7f77fc023210 con 0x5640603f1eb0
2016-06-02 14:09:22.937574 7f77b2ffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2568 9.9063c57f .dir.62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call rgw.bucket_complete_op] snapc 0=[] ack+ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77842cba00 con 0x5640603f1eb0
2016-06-02 14:09:22.937605 7f77b2ffd700 20 RGWDataChangesLog::add_entry() bucket.name=lotr shard_id=-1 now=2016-06-02 14:09:22.0.937604s cur_expiration=2016-06-02 14:09:33.0.428063s
2016-06-02 14:09:22.937835 7f7760ff9700 20 cr:s=0x7f773c707c60:op=0x7f773c389540:19RGWFetchRemoteObjCR: operate()
2016-06-02 14:09:22.937842 7f7760ff9700 20 cr:s=0x7f773c707c60:op=0x7f773c389540:19RGWFetchRemoteObjCR: operate()
2016-06-02 14:09:22.937843 7f7760ff9700 20 cr:s=0x7f773c707c60:op=0x7f773c389540:19RGWFetchRemoteObjCR: operate()
2016-06-02 14:09:22.937844 7f7760ff9700 20 cr:s=0x7f773c707c60:op=0x7f773c389540:19RGWFetchRemoteObjCR: operate()
2016-06-02 14:09:22.938138 7f7760ff9700 20 cr:s=0x7f773c707c60:op=0x7f773c55d3b0:26RGWBucketSyncSingleEntryCRINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE11rgw_obj_keyE: operate()
2016-06-02 14:09:22.938149 7f7760ff9700 20 abhi: deleting markeratroll5marker: 00000000468.1085.3
2016-06-02 14:09:22.938162 7f7760ff9700 20 cr:s=0x7f773c707c60:op=0x7f773c55d3b0:26RGWBucketSyncSingleEntryCRINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE11rgw_obj_keyE: operate()
2016-06-02 14:09:22.938165 7f7760ff9700 20 run: stack=0x7f773c707c60 is done
2016-06-02 14:09:22.938172 7f7760ff9700 20 cr:s=0x7f773c35b960:op=0x7f773c6bd190:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:22.938175 7f7760ff9700 20 collect(): s=0x7f773c35b960 stack=0x7f773c707c60 is complete
2016-06-02 14:09:22.938177 7f7760ff9700 20 cr:s=0x7f773c35b960:op=0x7f773c6bd190:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:22.938188 7f7760ff9700 20 cr:s=0x7f773c35b960:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:22.938193 7f7760ff9700 20 cr:s=0x7f773c35b960:op=0x7f773c708090:24RGWDataSyncSingleEntryCR: operate()
2016-06-02 14:09:22.938200 7f7760ff9700 20 run: stack=0x7f773c35b960 is done
2016-06-02 14:09:22.951147 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 877 ==== osd_op_reply(2568 .dir.62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call] v32'48 uv48 ack = 0) v7 ==== 169+0+0 (3958606177 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:22.951183 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 878 ==== osd_op_reply(2568 .dir.62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call] v32'48 uv48 ondisk = 0) v7 ==== 169+0+0 (4164626987 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:23.114931 7f77467fc700 20 process(): notifying datalog change, shard_id=36: lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1
2016-06-02 14:09:23.115013 7f77467fc700 20 cr:s=0x7f773001ad90:op=0x7f773000efd0:21RGWPostRESTResourceCRISt3mapIiSt3setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4lessIS7_ESaIS7_EES8_IiESaISt4pairIKiSB_EEEiE: operate()
2016-06-02 14:09:23.115018 7f77467fc700 20 cr:s=0x7f773001ad90:op=0x7f773000efd0:21RGWPostRESTResourceCRISt3mapIiSt3setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4lessIS7_ESaIS7_EES8_IiESaISt4pairIKiSB_EEEiE: operate()
2016-06-02 14:09:23.115120 7f77467fc700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:23 2016
2016-06-02 14:09:23.115123 7f77467fc700 20 > HTTP_DATE -> Thu Jun  2 12:09:23 2016
2016-06-02 14:09:23.115138 7f77467fc700 10 get_canon_resource(): dest=/admin/log
2016-06-02 14:09:23.115139 7f77467fc700 10 generated canonical header: POST


Thu Jun  2 12:09:23 2016
/admin/log
2016-06-02 14:09:23.115181 7f77467fc700 15 generated auth header: AWS 1555b35654ad1656d804:EQbmLHQFyGaeMfld+NmcUuotVJo=
2016-06-02 14:09:23.115190 7f77467fc700 20 sending request to http://localhost:8002/admin/log?type=data&notify&source-zone=62bc922d-f295-4067-ae36-e23e2f231aad&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:23.115205 7f77467fc700 20 register_request mgr=0x564060436fc0 req_data->id=2, easy_handle=0x7f773001f5c0
2016-06-02 14:09:23.115213 7f77467fc700 20 run: stack=0x7f773001ad90 is io blocked
2016-06-02 14:09:23.115239 7f7746ffd700 20 link_request req_data=0x7f7730016010 req_data->id=2, easy_handle=0x7f773001f5c0
2016-06-02 14:09:23.394437 7f7828ff9700  2 RGWDataChangesLog::ChangesRenewThread: start
2016-06-02 14:09:23.394527 7f7828ff9700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2569 5.f9325cdd data_log.36 [call log.add] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f7814006880 con 0x5640603e9110
2016-06-02 14:09:23.401986 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1180 ==== osd_op_reply(2569 data_log.36 [call] v32'1686 uv1686 ondisk = 0) v7 ==== 131+0+0 (789520816 0 0) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:23.402032 7f7828ff9700 20 RGWDataChangesLog::update_renewd() bucket_name=lotr shard_id=-1 expiration=2016-06-02 14:09:53.0.394478s
2016-06-02 14:09:23.581694 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:23.581735 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:23.581853 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:23 2016
2016-06-02 14:09:23.581856 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:23 2016
2016-06-02 14:09:23.581869 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:23.581870 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:23 2016
/admin/log/
2016-06-02 14:09:23.581903 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:mft5a4E6tNOXXmS6I3ROshN71kc=
2016-06-02 14:09:23.581922 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:23.581937 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=334, easy_handle=0x7f773c380450
2016-06-02 14:09:23.581944 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:23.581953 7f77477fe700 20 link_request req_data=0x7f773c073bd0 req_data->id=334, easy_handle=0x7f773c380450
2016-06-02 14:09:23.582907 7f77477fe700 10 receive_http_header
2016-06-02 14:09:23.582921 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:23.582923 7f77477fe700 10 receive_http_header
2016-06-02 14:09:23.582924 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001ee-00575021f3-3728-us-west
2016-06-02 14:09:23.582927 7f77477fe700 10 receive_http_header
2016-06-02 14:09:23.582927 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:23.582929 7f77477fe700 10 receive_http_header
2016-06-02 14:09:23.582929 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:23 GMT
2016-06-02 14:09:23.582931 7f77477fe700 10 receive_http_header
2016-06-02 14:09:23.582931 7f77477fe700 10 received header:
2016-06-02 14:09:23.582985 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:23.583121 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:23.583125 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869342.751510_3798.1 sync_marker.marker=1_1464869342.751510_3798.1
2016-06-02 14:09:23.583137 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869342.751510_3798.1 sync_marker.marker=1_1464869342.751510_3798.1
2016-06-02 14:09:23.583143 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:24.116930 7f7746ffd700 10 receive_http_header
2016-06-02 14:09:24.116939 7f7746ffd700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:24.116942 7f7746ffd700 10 receive_http_header
2016-06-02 14:09:24.116943 7f7746ffd700 10 received header:x-amz-request-id: tx0000000000000000001ed-00575021f3-3728-us-west
2016-06-02 14:09:24.116948 7f7746ffd700 10 receive_http_header
2016-06-02 14:09:24.116949 7f7746ffd700 10 received header:Content-Length: 0
2016-06-02 14:09:24.116951 7f7746ffd700 10 receive_http_header
2016-06-02 14:09:24.116952 7f7746ffd700 10 received header:Date: Thu, 02 Jun 2016 12:09:24 GMT
2016-06-02 14:09:24.116954 7f7746ffd700 10 receive_http_header
2016-06-02 14:09:24.116964 7f7746ffd700 10 received header:
2016-06-02 14:09:24.116993 7f77467fc700 20 cr:s=0x7f773001ad90:op=0x7f773000efd0:21RGWPostRESTResourceCRISt3mapIiSt3setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4lessIS7_ESaIS7_EES8_IiESaISt4pairIKiSB_EEEiE: operate()
2016-06-02 14:09:24.117380 7f77467fc700 20 cr:s=0x7f773001ad90:op=0x7f773000efd0:21RGWPostRESTResourceCRISt3mapIiSt3setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4lessIS7_ESaIS7_EES8_IiESaISt4pairIKiSB_EEEiE: operate()
2016-06-02 14:09:24.117385 7f77467fc700 20 cr:s=0x7f773001ad90:op=0x7f773000efd0:21RGWPostRESTResourceCRISt3mapIiSt3setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4lessIS7_ESaIS7_EES8_IiESaISt4pairIKiSB_EEEiE: operate()
2016-06-02 14:09:24.117387 7f77467fc700 20 cr:s=0x7f773001ad90:op=0x7f773000efd0:21RGWPostRESTResourceCRISt3mapIiSt3setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4lessIS7_ESaIS7_EES8_IiESaISt4pairIKiSB_EEEiE: operate()
2016-06-02 14:09:24.117391 7f77467fc700 20 run: stack=0x7f773001ad90 is done
2016-06-02 14:09:24.117660 7f770a7e4700 20 RGWEnv::set(): HTTP_HOST: localhost:8001
2016-06-02 14:09:24.117665 7f770a7e4700 20 RGWEnv::set(): HTTP_ACCEPT: */*
2016-06-02 14:09:24.117667 7f770a7e4700 20 RGWEnv::set(): HTTP_TRANSFER_ENCODING: chunked
2016-06-02 14:09:24.117668 7f770a7e4700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS 1555b35654ad1656d804:n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.117680 7f770a7e4700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.117680 7f770a7e4700 20 RGWEnv::set(): HTTP_EXPECT: 100-continue
2016-06-02 14:09:24.117681 7f770a7e4700 20 RGWEnv::set(): REQUEST_METHOD: GET
2016-06-02 14:09:24.117682 7f770a7e4700 20 RGWEnv::set(): REQUEST_URI: /admin/log/
2016-06-02 14:09:24.117682 7f770a7e4700 20 RGWEnv::set(): QUERY_STRING: type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.117683 7f770a7e4700 20 RGWEnv::set(): REMOTE_USER:
2016-06-02 14:09:24.117684 7f770a7e4700 20 RGWEnv::set(): SCRIPT_URI: /admin/log/
2016-06-02 14:09:24.117685 7f770a7e4700 20 RGWEnv::set(): SERVER_PORT: 8001
2016-06-02 14:09:24.117685 7f770a7e4700 20 HTTP_ACCEPT=*/*
2016-06-02 14:09:24.117686 7f770a7e4700 20 HTTP_AUTHORIZATION=AWS 1555b35654ad1656d804:n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.117686 7f770a7e4700 20 HTTP_DATE=Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.117686 7f770a7e4700 20 HTTP_EXPECT=100-continue
2016-06-02 14:09:24.117687 7f770a7e4700 20 HTTP_HOST=localhost:8001
2016-06-02 14:09:24.117687 7f770a7e4700 20 HTTP_TRANSFER_ENCODING=chunked
2016-06-02 14:09:24.117687 7f770a7e4700 20 QUERY_STRING=type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.117688 7f770a7e4700 20 REMOTE_USER=
2016-06-02 14:09:24.117688 7f770a7e4700 20 REQUEST_METHOD=GET
2016-06-02 14:09:24.117688 7f770a7e4700 20 REQUEST_URI=/admin/log/
2016-06-02 14:09:24.117689 7f770a7e4700 20 SCRIPT_URI=/admin/log/
2016-06-02 14:09:24.117689 7f770a7e4700 20 SERVER_PORT=8001
2016-06-02 14:09:24.117690 7f770a7e4700  1 ====== starting new request req=0x7f770a7de790 =====
2016-06-02 14:09:24.117701 7f770a7e4700  2 req 630:0.000011::GET /admin/log/::initializing for trans_id = tx000000000000000000276-00575021f4-5e2e-us-east-1
2016-06-02 14:09:24.117704 7f770a7e4700 10 host=localhost
2016-06-02 14:09:24.117706 7f770a7e4700 20 subdomain= domain=localhost in_hosted_domain=1 in_hosted_domain_s3website=0
2016-06-02 14:09:24.117734 7f770a7e4700 10 handler=14RGWHandler_Log
2016-06-02 14:09:24.117736 7f770a7e4700  2 req 630:0.000045::GET /admin/log/::getting op 0
2016-06-02 14:09:24.117738 7f770a7e4700 10 op=23RGWOp_DATALog_ShardInfo
2016-06-02 14:09:24.117739 7f770a7e4700  2 req 630:0.000049::GET /admin/log/:get_data_changes_log_shard_info:authorizing
2016-06-02 14:09:24.117756 7f770a7e4700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:24.117757 7f770a7e4700 10 auth_hdr:
GET


Thu Jun  2 12:09:24 2016
/admin/log/
2016-06-02 14:09:24.117784 7f770a7e4700 15 calculated digest=n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.117786 7f770a7e4700 15 auth_sign=n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.117787 7f770a7e4700 15 compare=0
2016-06-02 14:09:24.117788 7f770a7e4700 20 system request
2016-06-02 14:09:24.117790 7f770a7e4700  2 req 630:0.000100::GET /admin/log/:get_data_changes_log_shard_info:normalizing buckets and tenants
2016-06-02 14:09:24.117791 7f770a7e4700  2 req 630:0.000101::GET /admin/log/:get_data_changes_log_shard_info:init permissions
2016-06-02 14:09:24.117794 7f770a7e4700  2 req 630:0.000104::GET /admin/log/:get_data_changes_log_shard_info:recalculating target
2016-06-02 14:09:24.117794 7f770a7e4700  2 req 630:0.000104::GET /admin/log/:get_data_changes_log_shard_info:reading permissions
2016-06-02 14:09:24.117795 7f770a7e4700  2 req 630:0.000105::GET /admin/log/:get_data_changes_log_shard_info:init op
2016-06-02 14:09:24.117796 7f770a7e4700  2 req 630:0.000106::GET /admin/log/:get_data_changes_log_shard_info:verifying op mask
2016-06-02 14:09:24.117797 7f770a7e4700 20 required_mask= 0 user.op_mask=7
2016-06-02 14:09:24.117798 7f770a7e4700  2 req 630:0.000107::GET /admin/log/:get_data_changes_log_shard_info:verifying op permissions
2016-06-02 14:09:24.117799 7f770a7e4700  2 overriding permissions due to system operation
2016-06-02 14:09:24.117799 7f770a7e4700  2 req 630:0.000109::GET /admin/log/:get_data_changes_log_shard_info:verifying op params
2016-06-02 14:09:24.117800 7f770a7e4700  2 req 630:0.000110::GET /admin/log/:get_data_changes_log_shard_info:pre-executing
2016-06-02 14:09:24.117801 7f770a7e4700  2 req 630:0.000111::GET /admin/log/:get_data_changes_log_shard_info:executing
2016-06-02 14:09:24.117843 7f770a7e4700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2570 5.f9325cdd data_log.36 [call log.info] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f779c031650 con 0x5640603e9110
2016-06-02 14:09:24.118262 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1181 ==== osd_op_reply(2570 data_log.36 [call] v0'0 uv1686 ondisk = 0) v7 ==== 131+0+50 (2178453349 0 3348773465) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:24.118291 7f770a7e4700  2 req 630:0.000601::GET /admin/log/:get_data_changes_log_shard_info:completing
2016-06-02 14:09:24.118333 7f770a7e4700  2 req 630:0.000643::GET /admin/log/:get_data_changes_log_shard_info:op status=0
2016-06-02 14:09:24.118336 7f770a7e4700  2 req 630:0.000646::GET /admin/log/:get_data_changes_log_shard_info:http status=200
2016-06-02 14:09:24.118337 7f770a7e4700  1 ====== req done req=0x7f770a7de790 op status=0 http_status=200 ======
2016-06-02 14:09:24.118500 7f770a7e4700  1 civetweb: 0x7f779c003e30: 127.0.0.1 - - [02/Jun/2016:14:09:24 +0200] "GET /admin/log/ HTTP/1.1" 200 0 - -
2016-06-02 14:09:24.119106 7f770a7e4700 20 RGWEnv::set(): HTTP_HOST: localhost:8001
2016-06-02 14:09:24.119110 7f770a7e4700 20 RGWEnv::set(): HTTP_ACCEPT: */*
2016-06-02 14:09:24.119112 7f770a7e4700 20 RGWEnv::set(): HTTP_TRANSFER_ENCODING: chunked
2016-06-02 14:09:24.119113 7f770a7e4700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS 1555b35654ad1656d804:n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.119115 7f770a7e4700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.119116 7f770a7e4700 20 RGWEnv::set(): HTTP_EXPECT: 100-continue
2016-06-02 14:09:24.119116 7f770a7e4700 20 RGWEnv::set(): REQUEST_METHOD: GET
2016-06-02 14:09:24.119117 7f770a7e4700 20 RGWEnv::set(): REQUEST_URI: /admin/log/
2016-06-02 14:09:24.119118 7f770a7e4700 20 RGWEnv::set(): QUERY_STRING: type=data&id=36&marker=1_1464869343.428063_1682.1&extra-info=true&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.119119 7f770a7e4700 20 RGWEnv::set(): REMOTE_USER:
2016-06-02 14:09:24.119120 7f770a7e4700 20 RGWEnv::set(): SCRIPT_URI: /admin/log/
2016-06-02 14:09:24.119121 7f770a7e4700 20 RGWEnv::set(): SERVER_PORT: 8001
2016-06-02 14:09:24.119121 7f770a7e4700 20 HTTP_ACCEPT=*/*
2016-06-02 14:09:24.119122 7f770a7e4700 20 HTTP_AUTHORIZATION=AWS 1555b35654ad1656d804:n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.119123 7f770a7e4700 20 HTTP_DATE=Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.119123 7f770a7e4700 20 HTTP_EXPECT=100-continue
2016-06-02 14:09:24.119124 7f770a7e4700 20 HTTP_HOST=localhost:8001
2016-06-02 14:09:24.119124 7f770a7e4700 20 HTTP_TRANSFER_ENCODING=chunked
2016-06-02 14:09:24.119125 7f770a7e4700 20 QUERY_STRING=type=data&id=36&marker=1_1464869343.428063_1682.1&extra-info=true&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.119126 7f770a7e4700 20 REMOTE_USER=
2016-06-02 14:09:24.119126 7f770a7e4700 20 REQUEST_METHOD=GET
2016-06-02 14:09:24.119127 7f770a7e4700 20 REQUEST_URI=/admin/log/
2016-06-02 14:09:24.119127 7f770a7e4700 20 SCRIPT_URI=/admin/log/
2016-06-02 14:09:24.119128 7f770a7e4700 20 SERVER_PORT=8001
2016-06-02 14:09:24.119129 7f770a7e4700  1 ====== starting new request req=0x7f770a7de790 =====
2016-06-02 14:09:24.119137 7f770a7e4700  2 req 631:0.000008::GET /admin/log/::initializing for trans_id = tx000000000000000000277-00575021f4-5e2e-us-east-1
2016-06-02 14:09:24.119140 7f770a7e4700 10 host=localhost
2016-06-02 14:09:24.119141 7f770a7e4700 20 subdomain= domain=localhost in_hosted_domain=1 in_hosted_domain_s3website=0
2016-06-02 14:09:24.119159 7f770a7e4700 10 handler=14RGWHandler_Log
2016-06-02 14:09:24.119160 7f770a7e4700  2 req 631:0.000031::GET /admin/log/::getting op 0
2016-06-02 14:09:24.119163 7f770a7e4700 10 op=18RGWOp_DATALog_List
2016-06-02 14:09:24.119164 7f770a7e4700  2 req 631:0.000035::GET /admin/log/:list_data_changes_log:authorizing
2016-06-02 14:09:24.119178 7f770a7e4700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:24.119179 7f770a7e4700 10 auth_hdr:
GET


Thu Jun  2 12:09:24 2016
/admin/log/
2016-06-02 14:09:24.119201 7f770a7e4700 15 calculated digest=n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.119202 7f770a7e4700 15 auth_sign=n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.119202 7f770a7e4700 15 compare=0
2016-06-02 14:09:24.119203 7f770a7e4700 20 system request
2016-06-02 14:09:24.119206 7f770a7e4700  2 req 631:0.000077::GET /admin/log/:list_data_changes_log:normalizing buckets and tenants
2016-06-02 14:09:24.119207 7f770a7e4700  2 req 631:0.000078::GET /admin/log/:list_data_changes_log:init permissions
2016-06-02 14:09:24.119209 7f770a7e4700  2 req 631:0.000081::GET /admin/log/:list_data_changes_log:recalculating target
2016-06-02 14:09:24.119211 7f770a7e4700  2 req 631:0.000082::GET /admin/log/:list_data_changes_log:reading permissions
2016-06-02 14:09:24.119211 7f770a7e4700  2 req 631:0.000083::GET /admin/log/:list_data_changes_log:init op
2016-06-02 14:09:24.119213 7f770a7e4700  2 req 631:0.000084::GET /admin/log/:list_data_changes_log:verifying op mask
2016-06-02 14:09:24.119214 7f770a7e4700 20 required_mask= 0 user.op_mask=7
2016-06-02 14:09:24.119215 7f770a7e4700  2 req 631:0.000086::GET /admin/log/:list_data_changes_log:verifying op permissions
2016-06-02 14:09:24.119216 7f770a7e4700  2 overriding permissions due to system operation
2016-06-02 14:09:24.119217 7f770a7e4700  2 req 631:0.000088::GET /admin/log/:list_data_changes_log:verifying op params
2016-06-02 14:09:24.119218 7f770a7e4700  2 req 631:0.000089::GET /admin/log/:list_data_changes_log:pre-executing
2016-06-02 14:09:24.119219 7f770a7e4700  2 req 631:0.000090::GET /admin/log/:list_data_changes_log:executing
2016-06-02 14:09:24.119242 7f770a7e4700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2571 5.f9325cdd data_log.36 [call log.list] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f779c035790 con 0x5640603e9110
2016-06-02 14:09:24.119520 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1182 ==== osd_op_reply(2571 data_log.36 [call] v0'0 uv1686 ondisk = 0) v7 ==== 131+0+169 (4199196421 0 3831315810) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:24.119546 7f770a7e4700  2 req 631:0.000417::GET /admin/log/:list_data_changes_log:completing
2016-06-02 14:09:24.119581 7f770a7e4700  2 req 631:0.000452::GET /admin/log/:list_data_changes_log:op status=0
2016-06-02 14:09:24.119583 7f770a7e4700  2 req 631:0.000454::GET /admin/log/:list_data_changes_log:http status=200
2016-06-02 14:09:24.119584 7f770a7e4700  1 ====== req done req=0x7f770a7de790 op status=0 http_status=200 ======
2016-06-02 14:09:24.119710 7f770a7e4700  1 civetweb: 0x7f779c003e30: 127.0.0.1 - - [02/Jun/2016:14:09:24 +0200] "GET /admin/log/ HTTP/1.1" 200 0 - -
2016-06-02 14:09:24.131602 7f77097e2700 20 RGWEnv::set(): HTTP_HOST: localhost:8001
2016-06-02 14:09:24.131606 7f77097e2700 20 RGWEnv::set(): HTTP_ACCEPT: */*
2016-06-02 14:09:24.131608 7f77097e2700 20 RGWEnv::set(): HTTP_TRANSFER_ENCODING: chunked
2016-06-02 14:09:24.131610 7f77097e2700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS 1555b35654ad1656d804:F/uYrgDjBkvbbirJrlLzl+/BXKg=
2016-06-02 14:09:24.131611 7f77097e2700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.131612 7f77097e2700 20 RGWEnv::set(): HTTP_EXPECT: 100-continue
2016-06-02 14:09:24.131613 7f77097e2700 20 RGWEnv::set(): REQUEST_METHOD: GET
2016-06-02 14:09:24.131614 7f77097e2700 20 RGWEnv::set(): REQUEST_URI: /admin/log
2016-06-02 14:09:24.131616 7f77097e2700 20 RGWEnv::set(): QUERY_STRING: bucket-instance=lotr%3A62bc922d-f295-4067-ae36-e23e2f231aad.24099.1&format=json&marker=00000000016.45.3&type=bucket-index&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.131617 7f77097e2700 20 RGWEnv::set(): REMOTE_USER:
2016-06-02 14:09:24.131618 7f77097e2700 20 RGWEnv::set(): SCRIPT_URI: /admin/log
2016-06-02 14:09:24.131619 7f77097e2700 20 RGWEnv::set(): SERVER_PORT: 8001
2016-06-02 14:09:24.131620 7f77097e2700 20 HTTP_ACCEPT=*/*
2016-06-02 14:09:24.131621 7f77097e2700 20 HTTP_AUTHORIZATION=AWS 1555b35654ad1656d804:F/uYrgDjBkvbbirJrlLzl+/BXKg=
2016-06-02 14:09:24.131621 7f77097e2700 20 HTTP_DATE=Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.131622 7f77097e2700 20 HTTP_EXPECT=100-continue
2016-06-02 14:09:24.131622 7f77097e2700 20 HTTP_HOST=localhost:8001
2016-06-02 14:09:24.131623 7f77097e2700 20 HTTP_TRANSFER_ENCODING=chunked
2016-06-02 14:09:24.131623 7f77097e2700 20 QUERY_STRING=bucket-instance=lotr%3A62bc922d-f295-4067-ae36-e23e2f231aad.24099.1&format=json&marker=00000000016.45.3&type=bucket-index&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.131624 7f77097e2700 20 REMOTE_USER=
2016-06-02 14:09:24.131625 7f77097e2700 20 REQUEST_METHOD=GET
2016-06-02 14:09:24.131625 7f77097e2700 20 REQUEST_URI=/admin/log
2016-06-02 14:09:24.131626 7f77097e2700 20 SCRIPT_URI=/admin/log
2016-06-02 14:09:24.131626 7f77097e2700 20 SERVER_PORT=8001
2016-06-02 14:09:24.131627 7f77097e2700  1 ====== starting new request req=0x7f77097dc790 =====
2016-06-02 14:09:24.131637 7f77097e2700  2 req 632:0.000009::GET /admin/log::initializing for trans_id = tx000000000000000000278-00575021f4-5e2e-us-east-1
2016-06-02 14:09:24.131640 7f77097e2700 10 host=localhost
2016-06-02 14:09:24.131642 7f77097e2700 20 subdomain= domain=localhost in_hosted_domain=1 in_hosted_domain_s3website=0
2016-06-02 14:09:24.131664 7f77097e2700 10 handler=14RGWHandler_Log
2016-06-02 14:09:24.131666 7f77097e2700  2 req 632:0.000038::GET /admin/log::getting op 0
2016-06-02 14:09:24.131668 7f77097e2700 10 op=16RGWOp_BILog_List
2016-06-02 14:09:24.131669 7f77097e2700  2 req 632:0.000041::GET /admin/log:list_bucket_index_log:authorizing
2016-06-02 14:09:24.131684 7f77097e2700 10 get_canon_resource(): dest=/admin/log
2016-06-02 14:09:24.131685 7f77097e2700 10 auth_hdr:
GET


Thu Jun  2 12:09:24 2016
/admin/log
2016-06-02 14:09:24.131707 7f77097e2700 15 calculated digest=F/uYrgDjBkvbbirJrlLzl+/BXKg=
2016-06-02 14:09:24.131708 7f77097e2700 15 auth_sign=F/uYrgDjBkvbbirJrlLzl+/BXKg=
2016-06-02 14:09:24.131709 7f77097e2700 15 compare=0
2016-06-02 14:09:24.131710 7f77097e2700 20 system request
2016-06-02 14:09:24.131713 7f77097e2700  2 req 632:0.000085::GET /admin/log:list_bucket_index_log:normalizing buckets and tenants
2016-06-02 14:09:24.131714 7f77097e2700  2 req 632:0.000087::GET /admin/log:list_bucket_index_log:init permissions
2016-06-02 14:09:24.131716 7f77097e2700  2 req 632:0.000089::GET /admin/log:list_bucket_index_log:recalculating target
2016-06-02 14:09:24.131718 7f77097e2700  2 req 632:0.000090::GET /admin/log:list_bucket_index_log:reading permissions
2016-06-02 14:09:24.131719 7f77097e2700  2 req 632:0.000091::GET /admin/log:list_bucket_index_log:init op
2016-06-02 14:09:24.131720 7f77097e2700  2 req 632:0.000093::GET /admin/log:list_bucket_index_log:verifying op mask
2016-06-02 14:09:24.131721 7f77097e2700 20 required_mask= 0 user.op_mask=7
2016-06-02 14:09:24.131722 7f77097e2700  2 req 632:0.000095::GET /admin/log:list_bucket_index_log:verifying op permissions
2016-06-02 14:09:24.131723 7f77097e2700  2 overriding permissions due to system operation
2016-06-02 14:09:24.131724 7f77097e2700  2 req 632:0.000097::GET /admin/log:list_bucket_index_log:verifying op params
2016-06-02 14:09:24.131725 7f77097e2700  2 req 632:0.000098::GET /admin/log:list_bucket_index_log:pre-executing
2016-06-02 14:09:24.131726 7f77097e2700  2 req 632:0.000099::GET /admin/log:list_bucket_index_log:executing
2016-06-02 14:09:24.131730 7f77097e2700 20 reading from us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:24.131747 7f77097e2700 20 get_system_obj_state: rctx=0x7f77097db670 obj=us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 state=0x7f77ac02b0a8 s->prefetch_data=0
2016-06-02 14:09:24.131755 7f77097e2700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=22, cached=23)
2016-06-02 14:09:24.131762 7f77097e2700 20 get_system_obj_state: s->obj_tag was set empty
2016-06-02 14:09:24.131764 7f77097e2700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=17, cached=23)
2016-06-02 14:09:24.131782 7f77097e2700 20 list_bi_log_entries: lotr(@{i=us-east-1.rgw.buckets.index,e=us-east-1.rgw.buckets.non-ec}us-east-1.rgw.buckets.data[62bc922d-f295-4067-ae36-e23e2f231aad.24099.1]) marker 00000000016.45.3 shard_id=-1 max 1000
2016-06-02 14:09:24.131794 7f77097e2700 20 reading from us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:24.131799 7f77097e2700 20 get_system_obj_state: rctx=0x7f77097da600 obj=us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 state=0x7f77ac02e1f8 s->prefetch_data=0
2016-06-02 14:09:24.131805 7f77097e2700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=22, cached=23)
2016-06-02 14:09:24.131809 7f77097e2700 20 get_system_obj_state: s->obj_tag was set empty
2016-06-02 14:09:24.131813 7f77097e2700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=17, cached=23)
2016-06-02 14:09:24.131849 7f77097e2700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2572 9.9063c57f .dir.62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call rgw.bi_log_list] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77ac02e620 con 0x5640603f1eb0
2016-06-02 14:09:24.132175 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 879 ==== osd_op_reply(2572 .dir.62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call] v0'0 uv48 ondisk = 0) v7 ==== 169+0+233 (755583542 0 517292466) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:24.132340 7f77097e2700  2 req 632:0.000712::GET /admin/log:list_bucket_index_log:completing
2016-06-02 14:09:24.132377 7f77097e2700  2 req 632:0.000750::GET /admin/log:list_bucket_index_log:op status=0
2016-06-02 14:09:24.132380 7f77097e2700  2 req 632:0.000753::GET /admin/log:list_bucket_index_log:http status=200
2016-06-02 14:09:24.132383 7f77097e2700  1 ====== req done req=0x7f77097dc790 op status=0 http_status=200 ======
2016-06-02 14:09:24.132512 7f77097e2700  1 civetweb: 0x7f77ac002420: 127.0.0.1 - - [02/Jun/2016:14:09:24 +0200] "GET /admin/log HTTP/1.1" 200 0 - -
2016-06-02 14:09:24.132898 7f77097e2700 20 RGWEnv::set(): HTTP_HOST: localhost:8001
2016-06-02 14:09:24.132903 7f77097e2700 20 RGWEnv::set(): HTTP_ACCEPT: */*
2016-06-02 14:09:24.132904 7f77097e2700 20 RGWEnv::set(): HTTP_TRANSFER_ENCODING: chunked
2016-06-02 14:09:24.132906 7f77097e2700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS 1555b35654ad1656d804:F/uYrgDjBkvbbirJrlLzl+/BXKg=
2016-06-02 14:09:24.132917 7f77097e2700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.132918 7f77097e2700 20 RGWEnv::set(): HTTP_EXPECT: 100-continue
2016-06-02 14:09:24.132919 7f77097e2700 20 RGWEnv::set(): REQUEST_METHOD: GET
2016-06-02 14:09:24.132920 7f77097e2700 20 RGWEnv::set(): REQUEST_URI: /admin/log
2016-06-02 14:09:24.132921 7f77097e2700 20 RGWEnv::set(): QUERY_STRING: bucket-instance=lotr%3A62bc922d-f295-4067-ae36-e23e2f231aad.24099.1&format=json&marker=00000000018.47.3&type=bucket-index&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.132922 7f77097e2700 20 RGWEnv::set(): REMOTE_USER:
2016-06-02 14:09:24.132923 7f77097e2700 20 RGWEnv::set(): SCRIPT_URI: /admin/log
2016-06-02 14:09:24.132924 7f77097e2700 20 RGWEnv::set(): SERVER_PORT: 8001
2016-06-02 14:09:24.132925 7f77097e2700 20 HTTP_ACCEPT=*/*
2016-06-02 14:09:24.132925 7f77097e2700 20 HTTP_AUTHORIZATION=AWS 1555b35654ad1656d804:F/uYrgDjBkvbbirJrlLzl+/BXKg=
2016-06-02 14:09:24.132926 7f77097e2700 20 HTTP_DATE=Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.132927 7f77097e2700 20 HTTP_EXPECT=100-continue
2016-06-02 14:09:24.132927 7f77097e2700 20 HTTP_HOST=localhost:8001
2016-06-02 14:09:24.132928 7f77097e2700 20 HTTP_TRANSFER_ENCODING=chunked
2016-06-02 14:09:24.132928 7f77097e2700 20 QUERY_STRING=bucket-instance=lotr%3A62bc922d-f295-4067-ae36-e23e2f231aad.24099.1&format=json&marker=00000000018.47.3&type=bucket-index&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.132929 7f77097e2700 20 REMOTE_USER=
2016-06-02 14:09:24.132929 7f77097e2700 20 REQUEST_METHOD=GET
2016-06-02 14:09:24.132930 7f77097e2700 20 REQUEST_URI=/admin/log
2016-06-02 14:09:24.132930 7f77097e2700 20 SCRIPT_URI=/admin/log
2016-06-02 14:09:24.132931 7f77097e2700 20 SERVER_PORT=8001
2016-06-02 14:09:24.132932 7f77097e2700  1 ====== starting new request req=0x7f77097dc790 =====
2016-06-02 14:09:24.132950 7f77097e2700  2 req 633:0.000018::GET /admin/log::initializing for trans_id = tx000000000000000000279-00575021f4-5e2e-us-east-1
2016-06-02 14:09:24.132952 7f77097e2700 10 host=localhost
2016-06-02 14:09:24.132953 7f77097e2700 20 subdomain= domain=localhost in_hosted_domain=1 in_hosted_domain_s3website=0
2016-06-02 14:09:24.132982 7f77097e2700 10 handler=14RGWHandler_Log
2016-06-02 14:09:24.132983 7f77097e2700  2 req 633:0.000051::GET /admin/log::getting op 0
2016-06-02 14:09:24.132985 7f77097e2700 10 op=16RGWOp_BILog_List
2016-06-02 14:09:24.132986 7f77097e2700  2 req 633:0.000054::GET /admin/log:list_bucket_index_log:authorizing
2016-06-02 14:09:24.132997 7f77097e2700 10 get_canon_resource(): dest=/admin/log
2016-06-02 14:09:24.132997 7f77097e2700 10 auth_hdr:
GET


Thu Jun  2 12:09:24 2016
/admin/log
2016-06-02 14:09:24.133012 7f77097e2700 15 calculated digest=F/uYrgDjBkvbbirJrlLzl+/BXKg=
2016-06-02 14:09:24.133015 7f77097e2700 15 auth_sign=F/uYrgDjBkvbbirJrlLzl+/BXKg=
2016-06-02 14:09:24.133015 7f77097e2700 15 compare=0
2016-06-02 14:09:24.133016 7f77097e2700 20 system request
2016-06-02 14:09:24.133018 7f77097e2700  2 req 633:0.000086::GET /admin/log:list_bucket_index_log:normalizing buckets and tenants
2016-06-02 14:09:24.133020 7f77097e2700  2 req 633:0.000088::GET /admin/log:list_bucket_index_log:init permissions
2016-06-02 14:09:24.133022 7f77097e2700  2 req 633:0.000090::GET /admin/log:list_bucket_index_log:recalculating target
2016-06-02 14:09:24.133024 7f77097e2700  2 req 633:0.000092::GET /admin/log:list_bucket_index_log:reading permissions
2016-06-02 14:09:24.133025 7f77097e2700  2 req 633:0.000093::GET /admin/log:list_bucket_index_log:init op
2016-06-02 14:09:24.133026 7f77097e2700  2 req 633:0.000094::GET /admin/log:list_bucket_index_log:verifying op mask
2016-06-02 14:09:24.133027 7f77097e2700 20 required_mask= 0 user.op_mask=7
2016-06-02 14:09:24.133028 7f77097e2700  2 req 633:0.000096::GET /admin/log:list_bucket_index_log:verifying op permissions
2016-06-02 14:09:24.133029 7f77097e2700  2 overriding permissions due to system operation
2016-06-02 14:09:24.133029 7f77097e2700  2 req 633:0.000097::GET /admin/log:list_bucket_index_log:verifying op params
2016-06-02 14:09:24.133030 7f77097e2700  2 req 633:0.000098::GET /admin/log:list_bucket_index_log:pre-executing
2016-06-02 14:09:24.133031 7f77097e2700  2 req 633:0.000099::GET /admin/log:list_bucket_index_log:executing
2016-06-02 14:09:24.133035 7f77097e2700 20 reading from us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:24.133049 7f77097e2700 20 get_system_obj_state: rctx=0x7f77097db670 obj=us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 state=0x7f77ac02bb08 s->prefetch_data=0
2016-06-02 14:09:24.133056 7f77097e2700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=22, cached=23)
2016-06-02 14:09:24.133060 7f77097e2700 20 get_system_obj_state: s->obj_tag was set empty
2016-06-02 14:09:24.133063 7f77097e2700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=17, cached=23)
2016-06-02 14:09:24.133082 7f77097e2700 20 list_bi_log_entries: lotr(@{i=us-east-1.rgw.buckets.index,e=us-east-1.rgw.buckets.non-ec}us-east-1.rgw.buckets.data[62bc922d-f295-4067-ae36-e23e2f231aad.24099.1]) marker 00000000018.47.3 shard_id=-1 max 1000
2016-06-02 14:09:24.133089 7f77097e2700 20 reading from us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:24.133095 7f77097e2700 20 get_system_obj_state: rctx=0x7f77097da600 obj=us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 state=0x7f77ac032cd8 s->prefetch_data=0
2016-06-02 14:09:24.133098 7f77097e2700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=22, cached=23)
2016-06-02 14:09:24.133100 7f77097e2700 20 get_system_obj_state: s->obj_tag was set empty
2016-06-02 14:09:24.133101 7f77097e2700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=17, cached=23)
2016-06-02 14:09:24.133144 7f77097e2700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2573 9.9063c57f .dir.62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call rgw.bi_log_list] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77ac033190 con 0x5640603f1eb0
2016-06-02 14:09:24.133426 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 880 ==== osd_op_reply(2573 .dir.62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call] v0'0 uv48 ondisk = 0) v7 ==== 169+0+11 (2484288499 0 1993775135) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:24.133558 7f77097e2700  2 req 633:0.000626::GET /admin/log:list_bucket_index_log:completing
2016-06-02 14:09:24.133578 7f77097e2700  2 req 633:0.000646::GET /admin/log:list_bucket_index_log:op status=0
2016-06-02 14:09:24.133581 7f77097e2700  2 req 633:0.000649::GET /admin/log:list_bucket_index_log:http status=200
2016-06-02 14:09:24.133583 7f77097e2700  1 ====== req done req=0x7f77097dc790 op status=0 http_status=200 ======
2016-06-02 14:09:24.133694 7f77097e2700  1 civetweb: 0x7f77ac002420: 127.0.0.1 - - [02/Jun/2016:14:09:24 +0200] "GET /admin/log HTTP/1.1" 200 0 - -
2016-06-02 14:09:24.135506 7f77037d6700 20 RGWEnv::set(): HTTP_HOST: localhost:8001
2016-06-02 14:09:24.135511 7f77037d6700 20 RGWEnv::set(): HTTP_ACCEPT: */*
2016-06-02 14:09:24.135512 7f77037d6700 20 RGWEnv::set(): HTTP_TRANSFER_ENCODING: chunked
2016-06-02 14:09:24.135514 7f77037d6700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS 1555b35654ad1656d804:kgvbOL2gAw3W9sZmKLbApD+XT+o=
2016-06-02 14:09:24.135515 7f77037d6700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.135516 7f77037d6700 20 RGWEnv::set(): HTTP_EXPECT: 100-continue
2016-06-02 14:09:24.135516 7f77037d6700 20 RGWEnv::set(): REQUEST_METHOD: GET
2016-06-02 14:09:24.135517 7f77037d6700 20 RGWEnv::set(): REQUEST_URI: /lotr/atroll5
2016-06-02 14:09:24.135518 7f77037d6700 20 RGWEnv::set(): QUERY_STRING: rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a&rgwx-prepend-metadata=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.135518 7f77037d6700 20 RGWEnv::set(): REMOTE_USER:
2016-06-02 14:09:24.135519 7f77037d6700 20 RGWEnv::set(): SCRIPT_URI: /lotr/atroll5
2016-06-02 14:09:24.135520 7f77037d6700 20 RGWEnv::set(): SERVER_PORT: 8001
2016-06-02 14:09:24.135520 7f77037d6700 20 HTTP_ACCEPT=*/*
2016-06-02 14:09:24.135521 7f77037d6700 20 HTTP_AUTHORIZATION=AWS 1555b35654ad1656d804:kgvbOL2gAw3W9sZmKLbApD+XT+o=
2016-06-02 14:09:24.135521 7f77037d6700 20 HTTP_DATE=Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.135521 7f77037d6700 20 HTTP_EXPECT=100-continue
2016-06-02 14:09:24.135522 7f77037d6700 20 HTTP_HOST=localhost:8001
2016-06-02 14:09:24.135522 7f77037d6700 20 HTTP_TRANSFER_ENCODING=chunked
2016-06-02 14:09:24.135522 7f77037d6700 20 QUERY_STRING=rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a&rgwx-prepend-metadata=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.135523 7f77037d6700 20 REMOTE_USER=
2016-06-02 14:09:24.135523 7f77037d6700 20 REQUEST_METHOD=GET
2016-06-02 14:09:24.135524 7f77037d6700 20 REQUEST_URI=/lotr/atroll5
2016-06-02 14:09:24.135524 7f77037d6700 20 SCRIPT_URI=/lotr/atroll5
2016-06-02 14:09:24.135524 7f77037d6700 20 SERVER_PORT=8001
2016-06-02 14:09:24.135525 7f77037d6700  1 ====== starting new request req=0x7f77037d0790 =====
2016-06-02 14:09:24.135532 7f77037d6700  2 req 634:0.000007::GET /lotr/atroll5::initializing for trans_id = tx00000000000000000027a-00575021f4-5e2e-us-east-1
2016-06-02 14:09:24.135534 7f77037d6700 10 host=localhost
2016-06-02 14:09:24.135535 7f77037d6700 20 subdomain= domain=localhost in_hosted_domain=1 in_hosted_domain_s3website=0
2016-06-02 14:09:24.135547 7f77037d6700 20 get_handler handler=22RGWHandler_REST_Obj_S3
2016-06-02 14:09:24.135549 7f77037d6700 10 handler=22RGWHandler_REST_Obj_S3
2016-06-02 14:09:24.135550 7f77037d6700  2 req 634:0.000024:s3:GET /lotr/atroll5::getting op 0
2016-06-02 14:09:24.135551 7f77037d6700 10 op=21RGWGetObj_ObjStore_S3
2016-06-02 14:09:24.135552 7f77037d6700  2 req 634:0.000027:s3:GET /lotr/atroll5:get_obj:authorizing
2016-06-02 14:09:24.135562 7f77037d6700 10 get_canon_resource(): dest=/lotr/atroll5
2016-06-02 14:09:24.135563 7f77037d6700 10 auth_hdr:
GET


Thu Jun  2 12:09:24 2016
/lotr/atroll5
2016-06-02 14:09:24.135578 7f77037d6700 15 calculated digest=kgvbOL2gAw3W9sZmKLbApD+XT+o=
2016-06-02 14:09:24.135579 7f77037d6700 15 auth_sign=kgvbOL2gAw3W9sZmKLbApD+XT+o=
2016-06-02 14:09:24.135579 7f77037d6700 15 compare=0
2016-06-02 14:09:24.135580 7f77037d6700 20 system request
2016-06-02 14:09:24.135581 7f77037d6700  2 req 634:0.000056:s3:GET /lotr/atroll5:get_obj:normalizing buckets and tenants
2016-06-02 14:09:24.135583 7f77037d6700 10 s->object=atroll5 s->bucket=lotr
2016-06-02 14:09:24.135584 7f77037d6700  2 req 634:0.000059:s3:GET /lotr/atroll5:get_obj:init permissions
2016-06-02 14:09:24.135597 7f77037d6700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>foo</ID><DisplayName>foo</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>foo</ID><DisplayName>foo</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2016-06-02 14:09:24.135607 7f77037d6700  2 req 634:0.000081:s3:GET /lotr/atroll5:get_obj:recalculating target
2016-06-02 14:09:24.135609 7f77037d6700  2 req 634:0.000083:s3:GET /lotr/atroll5:get_obj:reading permissions
2016-06-02 14:09:24.135625 7f77037d6700 20 get_obj_state: rctx=0x7f77037cf670 obj=lotr:atroll5 state=0x7f77e40117b8 s->prefetch_data=1
2016-06-02 14:09:24.135649 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2574 10.a8f56d53 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1_atroll5 [getxattrs,stat,read 0~524288] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e4008590 con 0x5640603f1eb0
2016-06-02 14:09:24.136321 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 881 ==== osd_op_reply(2574 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1_atroll5 [getxattrs,stat,read 0~524288] v0'0 uv3109 ondisk = 0) v7 ==== 256+0+525284 (3560344323 0 2617716570) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:24.136399 7f77037d6700 10 manifest: total_size = 98878656
2016-06-02 14:09:24.136404 7f77037d6700 20 get_obj_state: setting s->obj_tag to 91441c60-0ca1-4118-9685-990e04272142.14120.350
2016-06-02 14:09:24.136407 7f77037d6700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>foo</ID><DisplayName>foo</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>foo</ID><DisplayName>foo</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2016-06-02 14:09:24.136412 7f77037d6700  2 req 634:0.000887:s3:GET /lotr/atroll5:get_obj:init op
2016-06-02 14:09:24.136415 7f77037d6700  2 req 634:0.000890:s3:GET /lotr/atroll5:get_obj:verifying op mask
2016-06-02 14:09:24.136416 7f77037d6700 20 required_mask= 1 user.op_mask=7
2016-06-02 14:09:24.136417 7f77037d6700  2 req 634:0.000891:s3:GET /lotr/atroll5:get_obj:verifying op permissions
2016-06-02 14:09:24.136422 7f77037d6700  5 Searching permissions for uid=zone.user mask=49
2016-06-02 14:09:24.136423 7f77037d6700  5 Permissions for user not found
2016-06-02 14:09:24.136423 7f77037d6700  5 Searching permissions for group=1 mask=49
2016-06-02 14:09:24.136424 7f77037d6700  5 Permissions for group not found
2016-06-02 14:09:24.136425 7f77037d6700  5 Searching permissions for group=2 mask=49
2016-06-02 14:09:24.136425 7f77037d6700  5 Permissions for group not found
2016-06-02 14:09:24.136425 7f77037d6700  5 Getting permissions id=zone.user owner=foo perm=0
2016-06-02 14:09:24.136426 7f77037d6700 10  uid=zone.user requested perm (type)=1, policy perm=0, user_perm_mask=15, acl perm=0
2016-06-02 14:09:24.136427 7f77037d6700  5 Searching permissions for uid=zone.user mask=48
2016-06-02 14:09:24.136427 7f77037d6700  5 Permissions for user not found
2016-06-02 14:09:24.136428 7f77037d6700  5 Searching permissions for group=1 mask=48
2016-06-02 14:09:24.136428 7f77037d6700  5 Permissions for group not found
2016-06-02 14:09:24.136428 7f77037d6700  5 Searching permissions for group=2 mask=48
2016-06-02 14:09:24.136429 7f77037d6700  5 Permissions for group not found
2016-06-02 14:09:24.136429 7f77037d6700  5 Getting permissions id=zone.user owner=foo perm=0
2016-06-02 14:09:24.136430 7f77037d6700 10  uid=zone.user requested perm (type)=16, policy perm=0, user_perm_mask=16, acl perm=0
2016-06-02 14:09:24.136431 7f77037d6700  2 overriding permissions due to system operation
2016-06-02 14:09:24.136431 7f77037d6700  2 req 634:0.000906:s3:GET /lotr/atroll5:get_obj:verifying op params
2016-06-02 14:09:24.136432 7f77037d6700  2 req 634:0.000907:s3:GET /lotr/atroll5:get_obj:pre-executing
2016-06-02 14:09:24.136443 7f77037d6700  2 req 634:0.000918:s3:GET /lotr/atroll5:get_obj:executing
2016-06-02 14:09:24.136459 7f77037d6700 20 get_obj_state: rctx=0x7f77037cf670 obj=lotr:atroll5 state=0x7f77e40117b8 s->prefetch_data=1
2016-06-02 14:09:24.136466 7f77037d6700 20 Read xattr: user.rgw.acl
2016-06-02 14:09:24.136466 7f77037d6700 20 Read xattr: user.rgw.content_type
2016-06-02 14:09:24.136466 7f77037d6700 20 Read xattr: user.rgw.etag
2016-06-02 14:09:24.136467 7f77037d6700 20 Read xattr: user.rgw.idtag
2016-06-02 14:09:24.136467 7f77037d6700 20 Read xattr: user.rgw.manifest
2016-06-02 14:09:24.136467 7f77037d6700 20 Read xattr: user.rgw.pg_ver
2016-06-02 14:09:24.136468 7f77037d6700 20 Read xattr: user.rgw.source_zone
2016-06-02 14:09:24.136477 7f77037d6700 20 get_obj_state: rctx=0x7f77037cf670 obj=lotr:atroll5 state=0x7f77e40117b8 s->prefetch_data=1
2016-06-02 14:09:24.136807 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_1 obj-ofs=524288 read_ofs=0 len=4194304
2016-06-02 14:09:24.136820 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2575 10.e37cf6b7 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_1 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e40131f0 con 0x5640603e9110
2016-06-02 14:09:24.136830 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.136835 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.136836 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=4718592 stripe_ofs=4718592 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.136841 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_2 obj-ofs=4718592 read_ofs=0 len=4194304
2016-06-02 14:09:24.136851 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2576 10.348f44ba 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_2 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e4019650 con 0x5640603ecb10
2016-06-02 14:09:24.136862 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.136866 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.136867 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=8912896 stripe_ofs=8912896 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.136873 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_3 obj-ofs=8912896 read_ofs=0 len=4194304
2016-06-02 14:09:24.136883 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2577 10.a6c32fe3 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_3 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e401a5e0 con 0x5640603f1eb0
2016-06-02 14:09:24.136892 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.136896 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.136897 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=13107200 stripe_ofs=13107200 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.136902 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_4 obj-ofs=13107200 read_ofs=0 len=4194304
2016-06-02 14:09:24.136910 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2578 10.13cc37d8 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_4 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e400f970 con 0x5640603f1eb0
2016-06-02 14:09:24.136917 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.136920 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.136921 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=17301504 stripe_ofs=17301504 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.144427 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 654 ==== osd_op_reply(2576 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_2 [read 0~4194304] v0'0 uv3143 ondisk = 0) v7 ==== 207+0+4194304 (2367705719 0 828494437) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:24.144499 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=4718592 len=4194304
2016-06-02 14:09:24.144533 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_5 obj-ofs=17301504 read_ofs=0 len=4194304
2016-06-02 14:09:24.144571 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2579 10.58ca6322 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_5 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e400f970 con 0x5640603ecb10
2016-06-02 14:09:24.144585 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.144601 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.144602 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=21495808 stripe_ofs=21495808 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.144936 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1183 ==== osd_op_reply(2575 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_1 [read 0~4194304] v0'0 uv3147 ondisk = 0) v7 ==== 207+0+4194304 (3051763062 0 1627770484) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:24.144972 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=524288 len=4194304
2016-06-02 14:09:24.144991 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_6 obj-ofs=21495808 read_ofs=0 len=4194304
2016-06-02 14:09:24.145007 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2580 10.19ff8d9d 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_6 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e4010770 con 0x5640603f1eb0
2016-06-02 14:09:24.145021 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.145796 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 882 ==== osd_op_reply(2578 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_4 [read 0~4194304] v0'0 uv2961 ondisk = 0) v7 ==== 207+0+4194304 (3197539961 0 1622225404) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:24.148442 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.148471 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=25690112 stripe_ofs=25690112 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.148557 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=13107200 len=4194304
2016-06-02 14:09:24.148820 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 883 ==== osd_op_reply(2577 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_3 [read 0~4194304] v0'0 uv3072 ondisk = 0) v7 ==== 207+0+4194304 (3761519233 0 3945593134) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:24.148879 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=8912896 len=4194304
2016-06-02 14:09:24.154080 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_7 obj-ofs=25690112 read_ofs=0 len=4194304
2016-06-02 14:09:24.154117 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2581 10.d858fc2a 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_7 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e4010770 con 0x5640603ecb10
2016-06-02 14:09:24.154591 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 884 ==== osd_op_reply(2580 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_6 [read 0~4194304] v0'0 uv3123 ondisk = 0) v7 ==== 207+0+4194304 (1494897817 0 419696575) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:24.154635 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=21495808 len=4194304
2016-06-02 14:09:24.155380 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 655 ==== osd_op_reply(2579 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_5 [read 0~4194304] v0'0 uv3152 ondisk = 0) v7 ==== 207+0+4194304 (1963370726 0 3157491970) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:24.156872 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.156900 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=17301504 len=4194304
2016-06-02 14:09:24.162586 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 656 ==== osd_op_reply(2581 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_7 [read 0~4194304] v0'0 uv3161 ondisk = 0) v7 ==== 207+0+4194304 (1699315637 0 600769771) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:24.162626 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=25690112 len=4194304
2016-06-02 14:09:24.213700 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.213710 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=29884416 stripe_ofs=29884416 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.213728 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_8 obj-ofs=29884416 read_ofs=0 len=4194304
2016-06-02 14:09:24.213753 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2582 10.e4842747 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_8 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e4010770 con 0x5640603e9110
2016-06-02 14:09:24.213769 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.217979 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1184 ==== osd_op_reply(2582 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_8 [read 0~4194304] v0'0 uv3156 ondisk = 0) v7 ==== 207+0+4194304 (4241568915 0 811829995) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:24.218057 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=29884416 len=4194304
2016-06-02 14:09:24.377308 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.377319 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=34078720 stripe_ofs=34078720 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.377339 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_9 obj-ofs=34078720 read_ofs=0 len=4194304
2016-06-02 14:09:24.377364 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2583 10.61804a5b 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_9 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e409d2e0 con 0x5640603f1eb0
2016-06-02 14:09:24.377429 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.385129 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 885 ==== osd_op_reply(2583 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_9 [read 0~4194304] v0'0 uv3081 ondisk = 0) v7 ==== 207+0+4194304 (3557324891 0 1051509891) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:24.385173 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=34078720 len=4194304
2016-06-02 14:09:24.396619 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2584 2.4322fa9f notify.0 [watch ping cookie 94834492702224] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f8015850 con 0x5640603ecb10
2016-06-02 14:09:24.396660 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2585 2.88aa5c95 notify.2 [watch ping cookie 94834492706960] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f80162c0 con 0x5640603ecb10
2016-06-02 14:09:24.396678 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2586 2.f8c99aee notify.3 [watch ping cookie 94834492709872] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f8016dd0 con 0x5640603ecb10
2016-06-02 14:09:24.396732 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2587 2.a204812d notify.4 [watch ping cookie 94834492712752] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f80183c0 con 0x5640603ecb10
2016-06-02 14:09:24.396752 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2588 2.31099063 notify.5 [watch ping cookie 94834492715648] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f8018ac0 con 0x5640603ecb10
2016-06-02 14:09:24.396759 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2589 2.97c520d4 notify.6 [watch ping cookie 94834492718368] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f8019520 con 0x5640603ecb10
2016-06-02 14:09:24.396785 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2590 2.16dafda0 notify.1 [watch ping cookie 94834492703856] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f8019ba0 con 0x5640603f1eb0
2016-06-02 14:09:24.396818 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2591 2.84ada7c9 notify.7 [watch ping cookie 94834492721344] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f801a220 con 0x5640603f1eb0
2016-06-02 14:09:24.396865 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- ping magic: 0 v1 -- ?+0 0x7f77f80183c0 con 0x5640603ecb10
2016-06-02 14:09:24.396871 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- ping magic: 0 v1 -- ?+0 0x7f77f801a8a0 con 0x5640603f1eb0
2016-06-02 14:09:24.397158 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 886 ==== osd_op_reply(2590 notify.1 [watch ping cookie 94834492703856] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (689729908 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:24.397286 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 887 ==== osd_op_reply(2591 notify.7 [watch ping cookie 94834492721344] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1842238908 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:24.397370 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 657 ==== osd_op_reply(2585 notify.2 [watch ping cookie 94834492706960] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3592388915 0 0) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:24.397455 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 658 ==== osd_op_reply(2584 notify.0 [watch ping cookie 94834492702224] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (4158488181 0 0) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:24.397497 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 659 ==== osd_op_reply(2586 notify.3 [watch ping cookie 94834492709872] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1695028518 0 0) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:24.397514 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 660 ==== osd_op_reply(2587 notify.4 [watch ping cookie 94834492712752] v0'0 uv3 ondisk = 0) v7 ==== 128+0+0 (1823463381 0 0) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:24.397531 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 661 ==== osd_op_reply(2588 notify.5 [watch ping cookie 94834492715648] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3716607297 0 0) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:24.397550 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 662 ==== osd_op_reply(2589 notify.6 [watch ping cookie 94834492718368] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1447217987 0 0) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:24.438598 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.438608 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=38273024 stripe_ofs=38273024 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.438625 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_10 obj-ofs=38273024 read_ofs=0 len=4194304
2016-06-02 14:09:24.438661 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2592 10.365fe1ab 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_10 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e409db30 con 0x5640603f1eb0
2016-06-02 14:09:24.440890 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.442562 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 888 ==== osd_op_reply(2592 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_10 [read 0~4194304] v0'0 uv3090 ondisk = 0) v7 ==== 208+0+4194304 (1330586322 0 1798421877) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:24.442642 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=38273024 len=4194304
2016-06-02 14:09:24.501983 7f76e6f9d700  1 -- 127.0.0.1:0/2056988180 --> 127.0.0.1:6808/30507 -- osd_op(client.24113.0:10 1.fd9d5d69 realms.00b17c78-da61-464b-906a-81dbe28b899d.control [watch ping cookie 94834493713904] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77fc024770 con 0x5640604f3290
2016-06-02 14:09:24.502005 7f76e6f9d700  1 -- 127.0.0.1:0/2056988180 --> 127.0.0.1:6808/30507 -- ping magic: 0 v1 -- ?+0 0x7f77fc0146c0 con 0x5640604f3290
2016-06-02 14:09:24.502447 7f78281f2700  1 -- 127.0.0.1:0/2056988180 <== osd.2 127.0.0.1:6808/30507 10 ==== osd_op_reply(10 realms.00b17c78-da61-464b-906a-81dbe28b899d.control [watch ping cookie 94834493713904] v0'0 uv1 ondisk = 0) v7 ==== 171+0+0 (3084123279 0 0) 0x7f77e801def0 con 0x5640604f3290
2016-06-02 14:09:24.543142 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:24.543223 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:24.543290 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.543293 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.543297 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:24.543298 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:24 2016
/admin/log/
2016-06-02 14:09:24.543326 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.543334 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.543342 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=335, easy_handle=0x7f773c380450
2016-06-02 14:09:24.543349 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:24.543361 7f77477fe700 20 link_request req_data=0x7f773c18d3d0 req_data->id=335, easy_handle=0x7f773c380450
2016-06-02 14:09:24.544144 7f77477fe700 10 receive_http_header
2016-06-02 14:09:24.544148 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:24.544149 7f77477fe700 10 receive_http_header
2016-06-02 14:09:24.544150 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001f0-00575021f4-3728-us-west
2016-06-02 14:09:24.544153 7f77477fe700 10 receive_http_header
2016-06-02 14:09:24.544153 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:24.544154 7f77477fe700 10 receive_http_header
2016-06-02 14:09:24.544155 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:24 GMT
2016-06-02 14:09:24.544156 7f77477fe700 10 receive_http_header
2016-06-02 14:09:24.544156 7f77477fe700 10 received header:
2016-06-02 14:09:24.544202 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:24.544485 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:24.544489 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869342.751510_3798.1 sync_marker.marker=1_1464869342.751510_3798.1
2016-06-02 14:09:24.544491 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869342.751510_3798.1 sync_marker.marker=1_1464869342.751510_3798.1
2016-06-02 14:09:24.544497 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:24.546131 7f770a7e4700 20 RGWEnv::set(): HTTP_HOST: localhost:8001
2016-06-02 14:09:24.546137 7f770a7e4700 20 RGWEnv::set(): HTTP_ACCEPT: */*
2016-06-02 14:09:24.546138 7f770a7e4700 20 RGWEnv::set(): HTTP_TRANSFER_ENCODING: chunked
2016-06-02 14:09:24.546140 7f770a7e4700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS 1555b35654ad1656d804:n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.546143 7f770a7e4700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.546144 7f770a7e4700 20 RGWEnv::set(): HTTP_EXPECT: 100-continue
2016-06-02 14:09:24.546144 7f770a7e4700 20 RGWEnv::set(): REQUEST_METHOD: GET
2016-06-02 14:09:24.546145 7f770a7e4700 20 RGWEnv::set(): REQUEST_URI: /admin/log/
2016-06-02 14:09:24.546146 7f770a7e4700 20 RGWEnv::set(): QUERY_STRING: type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.546147 7f770a7e4700 20 RGWEnv::set(): REMOTE_USER:
2016-06-02 14:09:24.546147 7f770a7e4700 20 RGWEnv::set(): SCRIPT_URI: /admin/log/
2016-06-02 14:09:24.546148 7f770a7e4700 20 RGWEnv::set(): SERVER_PORT: 8001
2016-06-02 14:09:24.546149 7f770a7e4700 20 HTTP_ACCEPT=*/*
2016-06-02 14:09:24.546149 7f770a7e4700 20 HTTP_AUTHORIZATION=AWS 1555b35654ad1656d804:n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.546150 7f770a7e4700 20 HTTP_DATE=Thu Jun  2 12:09:24 2016
2016-06-02 14:09:24.546150 7f770a7e4700 20 HTTP_EXPECT=100-continue
2016-06-02 14:09:24.546150 7f770a7e4700 20 HTTP_HOST=localhost:8001
2016-06-02 14:09:24.546151 7f770a7e4700 20 HTTP_TRANSFER_ENCODING=chunked
2016-06-02 14:09:24.546151 7f770a7e4700 20 QUERY_STRING=type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:24.546151 7f770a7e4700 20 REMOTE_USER=
2016-06-02 14:09:24.546152 7f770a7e4700 20 REQUEST_METHOD=GET
2016-06-02 14:09:24.546152 7f770a7e4700 20 REQUEST_URI=/admin/log/
2016-06-02 14:09:24.546152 7f770a7e4700 20 SCRIPT_URI=/admin/log/
2016-06-02 14:09:24.546153 7f770a7e4700 20 SERVER_PORT=8001
2016-06-02 14:09:24.546154 7f770a7e4700  1 ====== starting new request req=0x7f770a7de790 =====
2016-06-02 14:09:24.546163 7f770a7e4700  2 req 635:0.000010::GET /admin/log/::initializing for trans_id = tx00000000000000000027b-00575021f4-5e2e-us-east-1
2016-06-02 14:09:24.546166 7f770a7e4700 10 host=localhost
2016-06-02 14:09:24.546169 7f770a7e4700 20 subdomain= domain=localhost in_hosted_domain=1 in_hosted_domain_s3website=0
2016-06-02 14:09:24.546185 7f770a7e4700 10 handler=14RGWHandler_Log
2016-06-02 14:09:24.546186 7f770a7e4700  2 req 635:0.000032::GET /admin/log/::getting op 0
2016-06-02 14:09:24.546189 7f770a7e4700 10 op=23RGWOp_DATALog_ShardInfo
2016-06-02 14:09:24.546190 7f770a7e4700  2 req 635:0.000036::GET /admin/log/:get_data_changes_log_shard_info:authorizing
2016-06-02 14:09:24.546204 7f770a7e4700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:24.546205 7f770a7e4700 10 auth_hdr:
GET


Thu Jun  2 12:09:24 2016
/admin/log/
2016-06-02 14:09:24.546225 7f770a7e4700 15 calculated digest=n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.546225 7f770a7e4700 15 auth_sign=n6KhFOcNmaQimXxzRxlxFYI/8SU=
2016-06-02 14:09:24.546226 7f770a7e4700 15 compare=0
2016-06-02 14:09:24.546227 7f770a7e4700 20 system request
2016-06-02 14:09:24.546229 7f770a7e4700  2 req 635:0.000075::GET /admin/log/:get_data_changes_log_shard_info:normalizing buckets and tenants
2016-06-02 14:09:24.546230 7f770a7e4700  2 req 635:0.000076::GET /admin/log/:get_data_changes_log_shard_info:init permissions
2016-06-02 14:09:24.546232 7f770a7e4700  2 req 635:0.000078::GET /admin/log/:get_data_changes_log_shard_info:recalculating target
2016-06-02 14:09:24.546233 7f770a7e4700  2 req 635:0.000079::GET /admin/log/:get_data_changes_log_shard_info:reading permissions
2016-06-02 14:09:24.546234 7f770a7e4700  2 req 635:0.000080::GET /admin/log/:get_data_changes_log_shard_info:init op
2016-06-02 14:09:24.546235 7f770a7e4700  2 req 635:0.000081::GET /admin/log/:get_data_changes_log_shard_info:verifying op mask
2016-06-02 14:09:24.546236 7f770a7e4700 20 required_mask= 0 user.op_mask=7
2016-06-02 14:09:24.546236 7f770a7e4700  2 req 635:0.000083::GET /admin/log/:get_data_changes_log_shard_info:verifying op permissions
2016-06-02 14:09:24.546237 7f770a7e4700  2 overriding permissions due to system operation
2016-06-02 14:09:24.546238 7f770a7e4700  2 req 635:0.000084::GET /admin/log/:get_data_changes_log_shard_info:verifying op params
2016-06-02 14:09:24.546238 7f770a7e4700  2 req 635:0.000085::GET /admin/log/:get_data_changes_log_shard_info:pre-executing
2016-06-02 14:09:24.546239 7f770a7e4700  2 req 635:0.000085::GET /admin/log/:get_data_changes_log_shard_info:executing
2016-06-02 14:09:24.546284 7f770a7e4700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2593 5.f9325cdd data_log.36 [call log.info] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f779c039430 con 0x5640603e9110
2016-06-02 14:09:24.546700 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1185 ==== osd_op_reply(2593 data_log.36 [call] v0'0 uv1686 ondisk = 0) v7 ==== 131+0+50 (2178453349 0 3348773465) 0x7f780c020f60 con 0x5640603e9110
2016-06-02 14:09:24.546750 7f770a7e4700  2 req 635:0.000596::GET /admin/log/:get_data_changes_log_shard_info:completing
2016-06-02 14:09:24.546799 7f770a7e4700  2 req 635:0.000645::GET /admin/log/:get_data_changes_log_shard_info:op status=0
2016-06-02 14:09:24.546802 7f770a7e4700  2 req 635:0.000648::GET /admin/log/:get_data_changes_log_shard_info:http status=200
2016-06-02 14:09:24.546803 7f770a7e4700  1 ====== req done req=0x7f770a7de790 op status=0 http_status=200 ======
2016-06-02 14:09:24.546823 7f770a7e4700  1 civetweb: 0x7f779c003e30: 127.0.0.1 - - [02/Jun/2016:14:09:24 +0200] "GET /admin/log/ HTTP/1.1" 200 0 - -
2016-06-02 14:09:24.608439 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.608484 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=42467328 stripe_ofs=42467328 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.608513 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_11 obj-ofs=42467328 read_ofs=0 len=4194304
2016-06-02 14:09:24.608546 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2594 10.ad22732b 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_11 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e409e360 con 0x5640603f1eb0
2016-06-02 14:09:24.608587 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.616238 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.616670 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=46661632 stripe_ofs=46661632 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.616703 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_12 obj-ofs=46661632 read_ofs=0 len=4194304
2016-06-02 14:09:24.616743 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2595 10.47c0a1cf 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_12 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e409efd0 con 0x5640603e9110
2016-06-02 14:09:24.616763 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.616770 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.616771 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=50855936 stripe_ofs=50855936 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.616778 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_13 obj-ofs=50855936 read_ofs=0 len=4194304
2016-06-02 14:09:24.616793 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2596 10.260378d3 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_13 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e40a0000 con 0x5640603f1eb0
2016-06-02 14:09:24.619606 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.619676 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.619695 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=55050240 stripe_ofs=55050240 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.619738 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 obj-ofs=55050240 read_ofs=0 len=4194304
2016-06-02 14:09:24.619814 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2597 10.a468d144 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e40a0a30 con 0x5640603e9110
2016-06-02 14:09:24.619860 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.619880 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.619897 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=59244544 stripe_ofs=59244544 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.620687 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 889 ==== osd_op_reply(2594 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_11 [read 0~4194304] v0'0 uv3099 ondisk = 0) v7 ==== 208+0+4194304 (1206797462 0 2473665620) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:24.620732 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=42467328 len=4194304
2016-06-02 14:09:24.620764 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 obj-ofs=59244544 read_ofs=0 len=4194304
2016-06-02 14:09:24.620795 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2598 10.5ac080d4 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e40a1380 con 0x5640603e9110
2016-06-02 14:09:24.620812 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.622636 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1186 ==== osd_op_reply(2595 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_12 [read 0~4194304] v0'0 uv3165 ondisk = 0) v7 ==== 208+0+4194304 (160498624 0 139656153) 0x7f780c021410 con 0x5640603e9110
2016-06-02 14:09:24.623819 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=46661632 len=4194304
2016-06-02 14:09:24.628808 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 890 ==== osd_op_reply(2596 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_13 [read 0~4194304] v0'0 uv3108 ondisk = 0) v7 ==== 208+0+4194304 (2340260649 0 1469664682) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:24.628896 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=50855936 len=4194304
2016-06-02 14:09:24.629197 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1187 ==== osd_op_reply(2597 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_14 [read 0~4194304] v0'0 uv3162 ondisk = 0) v7 ==== 208+0+4194304 (90907866 0 994963540) 0x7f780c021410 con 0x5640603e9110
2016-06-02 14:09:24.629219 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=55050240 len=4194304
2016-06-02 14:09:24.630783 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1188 ==== osd_op_reply(2598 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_15 [read 0~4194304] v0'0 uv3171 ondisk = 0) v7 ==== 208+0+4194304 (2000525209 0 1814288981) 0x7f780c021410 con 0x5640603e9110
2016-06-02 14:09:24.630806 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=59244544 len=4194304
2016-06-02 14:09:24.720841 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:24.720854 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=63438848 stripe_ofs=63438848 part_ofs=0 rule->part_size=0
2016-06-02 14:09:24.720870 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 obj-ofs=63438848 read_ofs=0 len=4194304
2016-06-02 14:09:24.720889 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2599 10.8db93f86 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e409fc20 con 0x5640603ecb10
2016-06-02 14:09:24.720903 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:24.725951 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 663 ==== osd_op_reply(2599 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_16 [read 0~4194304] v0'0 uv2982 ondisk = 0) v7 ==== 208+0+4194304 (3864859257 0 1086314472) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:24.725996 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=63438848 len=4194304
2016-06-02 14:09:25.204930 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:25.204968 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=67633152 stripe_ofs=67633152 part_ofs=0 rule->part_size=0
2016-06-02 14:09:25.205004 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 obj-ofs=67633152 read_ofs=0 len=4194304
2016-06-02 14:09:25.205037 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2600 10.562651c7 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e40a19a0 con 0x5640603e9110
2016-06-02 14:09:25.205256 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:25.212088 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1189 ==== osd_op_reply(2600 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_17 [read 0~4194304] v0'0 uv3174 ondisk = 0) v7 ==== 208+0+4194304 (1792754134 0 1574641893) 0x7f780c021410 con 0x5640603e9110
2016-06-02 14:09:25.212141 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=67633152 len=4194304
2016-06-02 14:09:25.269654 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:25.269665 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=71827456 stripe_ofs=71827456 part_ofs=0 rule->part_size=0
2016-06-02 14:09:25.269681 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 obj-ofs=71827456 read_ofs=0 len=4194304
2016-06-02 14:09:25.269712 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2601 10.d2750535 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e40a22b0 con 0x5640603f1eb0
2016-06-02 14:09:25.273404 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:25.276638 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 891 ==== osd_op_reply(2601 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_18 [read 0~4194304] v0'0 uv3132 ondisk = 0) v7 ==== 208+0+4194304 (1329046381 0 2206506249) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:25.277432 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=71827456 len=4194304
2016-06-02 14:09:25.367893 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:25.367904 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=76021760 stripe_ofs=76021760 part_ofs=0 rule->part_size=0
2016-06-02 14:09:25.367920 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 obj-ofs=76021760 read_ofs=0 len=4194304
2016-06-02 14:09:25.367941 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2602 10.ea2c6d96 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e40a2ab0 con 0x5640603ecb10
2016-06-02 14:09:25.367956 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:25.374439 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 664 ==== osd_op_reply(2602 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_19 [read 0~4194304] v0'0 uv2991 ondisk = 0) v7 ==== 208+0+4194304 (782513266 0 3863633826) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:25.374477 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=76021760 len=4194304
2016-06-02 14:09:25.446982 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:25.446996 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=80216064 stripe_ofs=80216064 part_ofs=0 rule->part_size=0
2016-06-02 14:09:25.447023 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 obj-ofs=80216064 read_ofs=0 len=4194304
2016-06-02 14:09:25.447073 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2603 10.4990a3e5 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e40a32b0 con 0x5640603f1eb0
2016-06-02 14:09:25.447095 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:25.453894 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 892 ==== osd_op_reply(2603 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_20 [read 0~4194304] v0'0 uv3141 ondisk = 0) v7 ==== 208+0+4194304 (1106670307 0 3486073688) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:25.453938 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=80216064 len=4194304
2016-06-02 14:09:25.545167 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:25.545296 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:25.545371 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:25 2016
2016-06-02 14:09:25.545374 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:25 2016
2016-06-02 14:09:25.545388 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:25.545389 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:25 2016
/admin/log/
2016-06-02 14:09:25.545419 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:vbAINza7htKLE1v0E6GpuNJcUoQ=
2016-06-02 14:09:25.545437 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:25.545454 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=336, easy_handle=0x7f773c380450
2016-06-02 14:09:25.545459 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:25.545467 7f77477fe700 20 link_request req_data=0x7f773c0a0a10 req_data->id=336, easy_handle=0x7f773c380450
2016-06-02 14:09:25.546369 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.546373 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:25.546376 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.546376 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001f1-00575021f5-3728-us-west
2016-06-02 14:09:25.546381 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.546382 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:25.546383 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.546383 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:25 GMT
2016-06-02 14:09:25.546385 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.546385 7f77477fe700 10 received header:
2016-06-02 14:09:25.546454 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:25.546750 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:25.546754 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869342.751510_3798.1
2016-06-02 14:09:25.546788 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c459e70:27RGWReadRemoteDataLogShardCR: operate()
2016-06-02 14:09:25.546856 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:25 2016
2016-06-02 14:09:25.546858 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:25 2016
2016-06-02 14:09:25.546861 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:25.546861 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:25 2016
/admin/log/
2016-06-02 14:09:25.546887 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:vbAINza7htKLE1v0E6GpuNJcUoQ=
2016-06-02 14:09:25.546893 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&marker=1_1464869342.751510_3798.1&extra-info=true&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:25.546896 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=337, easy_handle=0x7f773c380450
2016-06-02 14:09:25.546901 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:25.546906 7f77477fe700 20 link_request req_data=0x7f773c17c520 req_data->id=337, easy_handle=0x7f773c380450
2016-06-02 14:09:25.547543 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.547546 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:25.547548 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.547549 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001f2-00575021f5-3728-us-west
2016-06-02 14:09:25.547553 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.547553 7f77477fe700 10 received header:Content-Length: 288
2016-06-02 14:09:25.547555 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.547556 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:25 GMT
2016-06-02 14:09:25.547558 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.547559 7f77477fe700 10 received header:
2016-06-02 14:09:25.585416 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c459e70:27RGWReadRemoteDataLogShardCR: operate()
2016-06-02 14:09:25.585925 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:25.585931 7f7760ff9700 20 incremental_sync:1068: shard_id=36 log_entry: 1_1464869364.852692_3801.1:2016-06-02 14:09:24.0.852692s:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:25.586003 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c2e2180:24RGWDataSyncSingleEntryCR: operate()
2016-06-02 14:09:25.586020 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:25.586023 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c26d9b0 is still running
2016-06-02 14:09:25.586024 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c333a30 is complete
2016-06-02 14:09:25.586025 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c0185c0 is complete
2016-06-02 14:09:25.586026 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c094510 is complete
2016-06-02 14:09:25.586027 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c053260 is complete
2016-06-02 14:09:25.586027 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c0afc00 is complete
2016-06-02 14:09:25.586029 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c55eb90 is complete
2016-06-02 14:09:25.586030 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c0d28c0 is complete
2016-06-02 14:09:25.586031 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c0e25a0 is complete
2016-06-02 14:09:25.586032 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c13b000 is complete
2016-06-02 14:09:25.586032 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c0de0c0 is complete
2016-06-02 14:09:25.586033 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c0ac590 is complete
2016-06-02 14:09:25.586034 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c0b00c0 is complete
2016-06-02 14:09:25.586035 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c195bc0 is complete
2016-06-02 14:09:25.586036 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c1636e0 is complete
2016-06-02 14:09:25.586037 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c14abd0 is complete
2016-06-02 14:09:25.586038 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c1706a0 is complete
2016-06-02 14:09:25.586039 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c0d8690 is complete
2016-06-02 14:09:25.586040 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c3934f0 is complete
2016-06-02 14:09:25.586042 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c5cab40 is complete
2016-06-02 14:09:25.586043 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c33b1f0 is complete
2016-06-02 14:09:25.586044 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c165a10 is complete
2016-06-02 14:09:25.586045 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c35b960 is complete
2016-06-02 14:09:25.586046 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c0a9c90 is complete
2016-06-02 14:09:25.586047 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c331700 is complete
2016-06-02 14:09:25.586048 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c0a44c0 is still running
2016-06-02 14:09:25.586051 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:25.586057 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:25.586059 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:25.586065 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:32RGWReadBucketSyncStatusCoroutine: operate()
2016-06-02 14:09:25.586071 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c671720:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:25.586073 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c671720:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:25.586089 7f7760ff9700 20 enqueued request req=0x7f773c2e06a0
2016-06-02 14:09:25.586090 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:25.586091 7f7760ff9700 20 req: 0x7f773c2e06a0
2016-06-02 14:09:25.586094 7f7760ff9700 20 run: stack=0x7f773c0a44c0 is io blocked
2016-06-02 14:09:25.586098 7f77c6ffd700 20 dequeued request req=0x7f773c2e06a0
2016-06-02 14:09:25.586102 7f77c6ffd700 20 RGWWQ: empty
2016-06-02 14:09:25.586112 7f77c6ffd700 10 cache get: name=us-east-1.rgw.log+bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=3, cached=11)
2016-06-02 14:09:25.586154 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c671720:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:25.586158 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c671720:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:25.586159 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c671720:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:25.586171 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c671720:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:25.586440 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:32RGWReadBucketSyncStatusCoroutine: operate()
2016-06-02 14:09:25.586458 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:25.586459 7f7760ff9700 20 operate(): sync status for bucket lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1: 2
2016-06-02 14:09:25.586465 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:25.586467 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:25.586473 7f7760ff9700 20 enqueued request req=0x7f773c2a7430
2016-06-02 14:09:25.586475 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:25.586476 7f7760ff9700 20 req: 0x7f773c2a7430
2016-06-02 14:09:25.586478 7f7760ff9700 20 run: stack=0x7f773c0a44c0 is io blocked
2016-06-02 14:09:25.586501 7f77b17fa700 20 dequeued request req=0x7f773c2a7430
2016-06-02 14:09:25.586510 7f77b17fa700 20 RGWWQ: empty
2016-06-02 14:09:25.586515 7f77b17fa700 20 reading from us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:25.586528 7f77b17fa700 20 get_system_obj_state: rctx=0x7f77b17f8560 obj=us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 state=0x7f778001e3e8 s->prefetch_data=0
2016-06-02 14:09:25.586545 7f77b17fa700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=22, cached=23)
2016-06-02 14:09:25.586549 7f77b17fa700 20 get_system_obj_state: s->obj_tag was set empty
2016-06-02 14:09:25.586551 7f77b17fa700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=17, cached=23)
2016-06-02 14:09:25.586592 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:25.586598 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:25.586599 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:25.586600 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:25.586837 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:25.586841 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:25.586842 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:25.586850 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:25.586875 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c0d79d0:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:25.586881 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:25.586882 7f7760ff9700 20 run: stack=0x7f773c0a44c0 is_blocked_by_stack()=0 is_sleeping=1 waiting_for_child()=0
2016-06-02 14:09:25.586884 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c3335e0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:25.586885 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c3335e0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:25.586893 7f7760ff9700 20 enqueued request req=0x7f773c1653e0
2016-06-02 14:09:25.586893 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:25.586894 7f7760ff9700 20 req: 0x7f773c1653e0
2016-06-02 14:09:25.586896 7f7760ff9700 20 run: stack=0x7f773c252ab0 is io blocked
2016-06-02 14:09:25.586902 7f7800ff9700 20 dequeued request req=0x7f773c1653e0
2016-06-02 14:09:25.586904 7f7800ff9700 20 RGWWQ: empty
2016-06-02 14:09:25.586947 7f7800ff9700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2604 5.944771f8 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77d801c170 con 0x5640603f1eb0
2016-06-02 14:09:25.752863 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 893 ==== osd_op_reply(2604 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call] v32'2379 uv2379 ondisk = 0) v7 ==== 225+0+0 (848001097 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:25.753927 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c3335e0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:25.753946 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c3335e0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:25.753949 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c3335e0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:25.753951 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c3335e0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:25.754343 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c0d79d0:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:25.754363 7f7760ff9700 20 run: stack=0x7f773c252ab0 is io blocked
2016-06-02 14:09:25.754365 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:25.754371 7f7760ff9700 20 abhi: starting a new RGWBucketIncSyncShardMarkerTrack with marker oidbucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:25.754373 7f7760ff9700 20 operate(): listing bilog for incremental sync
2016-06-02 14:09:25.754395 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c5d7ae0:23RGWListBucketIndexLogCR: operate()
2016-06-02 14:09:25.754409 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:25.754413 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:25.754611 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:25 2016
2016-06-02 14:09:25.754616 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:25 2016
2016-06-02 14:09:25.754623 7f7760ff9700 10 get_canon_resource(): dest=/admin/log
2016-06-02 14:09:25.754625 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:25 2016
/admin/log
2016-06-02 14:09:25.754679 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:BE37NGJExYsSzWr9hhTd2oEDSdE=
2016-06-02 14:09:25.754693 7f7760ff9700 20 sending request to http://localhost:8002/admin/log?bucket-instance=lotr%3A62bc922d-f295-4067-ae36-e23e2f231aad.24099.1&format=json&marker=00000000472.1089.3&type=bucket-index&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:25.754703 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=338, easy_handle=0x7f773c380450
2016-06-02 14:09:25.754717 7f7760ff9700 20 run: stack=0x7f773c0a44c0 is io blocked
2016-06-02 14:09:25.754735 7f77477fe700 20 link_request req_data=0x7f773c08f290 req_data->id=338, easy_handle=0x7f773c380450
2016-06-02 14:09:25.754858 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:25.754862 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=84410368 stripe_ofs=84410368 part_ofs=0 rule->part_size=0
2016-06-02 14:09:25.754881 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 obj-ofs=84410368 read_ofs=0 len=4194304
2016-06-02 14:09:25.754907 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2605 10.cc584f41 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e40a3ab0 con 0x5640603e9110
2016-06-02 14:09:25.754943 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:25.757795 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.757807 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:25.757810 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.757811 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001f3-00575021f5-3728-us-west
2016-06-02 14:09:25.757820 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.757820 7f77477fe700 10 received header:Content-Length: 2
2016-06-02 14:09:25.757823 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.757823 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:25 GMT
2016-06-02 14:09:25.757826 7f77477fe700 10 receive_http_header
2016-06-02 14:09:25.757826 7f77477fe700 10 received header:
2016-06-02 14:09:25.757914 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:25.758555 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:25.758565 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:25.758567 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:25.758575 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c5d7ae0:23RGWListBucketIndexLogCR: operate()
2016-06-02 14:09:25.758580 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:25.758583 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:25.758587 7f7760ff9700 20 run: stack=0x7f773c0a44c0 is_blocked_by_stack()=0 is_sleeping=0 waiting_for_child()=1
2016-06-02 14:09:25.758590 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c0d79d0:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:25.758901 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c3335e0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:25.758906 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c3335e0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:25.758932 7f7760ff9700 20 enqueued request req=0x7f773c6657a0
2016-06-02 14:09:25.758934 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:25.758935 7f7760ff9700 20 req: 0x7f773c6657a0
2016-06-02 14:09:25.758940 7f7760ff9700 20 run: stack=0x7f773c252ab0 is io blocked
2016-06-02 14:09:25.758949 7f78017fa700 20 dequeued request req=0x7f773c6657a0
2016-06-02 14:09:25.758950 7f78017fa700 20 RGWWQ: empty
2016-06-02 14:09:25.759013 7f78017fa700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2606 5.944771f8 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77e40a64c0 con 0x5640603f1eb0
2016-06-02 14:09:25.763427 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1190 ==== osd_op_reply(2605 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_21 [read 0~4194304] v0'0 uv2922 ondisk = 0) v7 ==== 208+0+4194304 (4067857709 0 1950557276) 0x7f780c021410 con 0x5640603e9110
2016-06-02 14:09:25.763467 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=84410368 len=4194304
2016-06-02 14:09:25.765504 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:25.765514 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=88604672 stripe_ofs=88604672 part_ofs=0 rule->part_size=0
2016-06-02 14:09:25.765530 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 obj-ofs=88604672 read_ofs=0 len=4194304
2016-06-02 14:09:25.765556 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2607 10.2109ab6d 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e40a6dd0 con 0x5640603f1eb0
2016-06-02 14:09:25.765569 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:25.774038 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 894 ==== osd_op_reply(2607 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_22 [read 0~4194304] v0'0 uv3150 ondisk = 0) v7 ==== 208+0+4194304 (2520255845 0 3529929581) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:25.774090 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=88604672 len=4194304
2016-06-02 14:09:25.799058 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 895 ==== osd_op_reply(2606 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call] v32'2380 uv2380 ondisk = 0) v7 ==== 225+0+0 (1722495014 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:25.799137 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c3335e0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:25.799154 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c3335e0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:25.799159 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c3335e0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:25.799161 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c3335e0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:25.799504 7f7760ff9700 20 cr:s=0x7f773c252ab0:op=0x7f773c0d79d0:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:25.799510 7f7760ff9700 20 run: stack=0x7f773c252ab0 is done
2016-06-02 14:09:25.799514 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:25.799518 7f7760ff9700 20 collect(): s=0x7f773c0a44c0 stack=0x7f773c252ab0 is complete
2016-06-02 14:09:25.799520 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:25.799531 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:25.799543 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c2e2180:24RGWDataSyncSingleEntryCR: operate()
2016-06-02 14:09:25.799550 7f7760ff9700 20 abhi: deleting markerlotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1marker: 1_1464869364.852692_3801.1
2016-06-02 14:09:25.799555 7f7760ff9700 20 store_marker(): updating marker marker_oid=datalog.sync-status.shard.91441c60-0ca1-4118-9685-990e04272142.36 marker=1_1464869364.852692_3801.1
2016-06-02 14:09:25.799580 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:21RGWSimpleRadosWriteCRI20rgw_data_sync_markerE: operate()
2016-06-02 14:09:25.799583 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:21RGWSimpleRadosWriteCRI20rgw_data_sync_markerE: operate()
2016-06-02 14:09:25.799605 7f7760ff9700 20 enqueued request req=0x7f773c146250
2016-06-02 14:09:25.799607 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:25.799608 7f7760ff9700 20 req: 0x7f773c146250
2016-06-02 14:09:25.799613 7f7760ff9700 20 run: stack=0x7f773c0a44c0 is io blocked
2016-06-02 14:09:25.799625 7f77aa7fc700 20 dequeued request req=0x7f773c146250
2016-06-02 14:09:25.799626 7f77aa7fc700 20 RGWWQ: empty
2016-06-02 14:09:25.799673 7f77aa7fc700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2608 5.c8d946cb datalog.sync-status.shard.91441c60-0ca1-4118-9685-990e04272142.36 [writefull 0~66] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f777000e530 con 0x5640603e9110
2016-06-02 14:09:25.841154 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1191 ==== osd_op_reply(2608 datalog.sync-status.shard.91441c60-0ca1-4118-9685-990e04272142.36 [writefull 0~66] v32'1887 uv1887 ondisk = 0) v7 ==== 185+0+0 (1216698362 0 0) 0x7f780c021410 con 0x5640603e9110
2016-06-02 14:09:25.841203 7f77aa7fc700 10 cache put: name=us-east-1.rgw.log+datalog.sync-status.shard.91441c60-0ca1-4118-9685-990e04272142.36 info.flags=1
2016-06-02 14:09:25.841211 7f77aa7fc700 10 moving us-east-1.rgw.log+datalog.sync-status.shard.91441c60-0ca1-4118-9685-990e04272142.36 to cache LRU end
2016-06-02 14:09:25.841234 7f77aa7fc700 10 distributing notification oid=notify.3 bl.length()=355
2016-06-02 14:09:25.841283 7f77aa7fc700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2609 2.f8c99aee notify.3 [notify cookie 140150956986400] snapc 0=[] ondisk+read+known_if_redirected e32) v7 -- ?+0 0x7f7770015df0 con 0x5640603ecb10
2016-06-02 14:09:25.841652 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 665 ==== watch-notify(notify (1) cookie 94834492709872 notify 137438953607 ret 0) v3 ==== 397+0+0 (2204707835 0 0) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:25.841688 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 666 ==== osd_op_reply(2609 notify.3 [notify cookie 140150956986400] v0'0 uv1 ondisk = 0) v7 ==== 128+0+8 (4049392337 0 427657195) 0x7f78040024c0 con 0x5640603ecb10
2016-06-02 14:09:25.841713 7f78297fa700 10 RGWWatcher::handle_notify()  notify_id 137438953607 cookie 94834492709872 notifier 24110 bl.length()=355
2016-06-02 14:09:25.841728 7f78297fa700 10 cache put: name=us-east-1.rgw.log+datalog.sync-status.shard.91441c60-0ca1-4118-9685-990e04272142.36 info.flags=1
2016-06-02 14:09:25.841732 7f78297fa700 10 moving us-east-1.rgw.log+datalog.sync-status.shard.91441c60-0ca1-4118-9685-990e04272142.36 to cache LRU end
2016-06-02 14:09:25.841751 7f78297fa700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2610 2.f8c99aee notify.3 [notify-ack 0~0] snapc 0=[] read+known_if_redirected e32) v7 -- ?+0 0x7f77e002bcd0 con 0x5640603ecb10
2016-06-02 14:09:25.842186 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 667 ==== watch-notify(notify_complete (2) cookie 140150956986400 notify 137438953607 ret 0) v3 ==== 42+0+28 (406505650 0 1747519784) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:25.842223 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 668 ==== osd_op_reply(2610 notify.3 [notify-ack 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (2497822551 0 0) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:25.842251 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:21RGWSimpleRadosWriteCRI20rgw_data_sync_markerE: operate()
2016-06-02 14:09:25.842256 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:21RGWSimpleRadosWriteCRI20rgw_data_sync_markerE: operate()
2016-06-02 14:09:25.842257 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:21RGWSimpleRadosWriteCRI20rgw_data_sync_markerE: operate()
2016-06-02 14:09:25.842259 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c136d60:21RGWSimpleRadosWriteCRI20rgw_data_sync_markerE: operate()
2016-06-02 14:09:25.842498 7f7760ff9700 20 cr:s=0x7f773c0a44c0:op=0x7f773c2e2180:24RGWDataSyncSingleEntryCR: operate()
2016-06-02 14:09:25.842503 7f7760ff9700 20 run: stack=0x7f773c0a44c0 is done
2016-06-02 14:09:25.943646 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:25.943658 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=92798976 stripe_ofs=92798976 part_ofs=0 rule->part_size=0
2016-06-02 14:09:25.943676 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 obj-ofs=92798976 read_ofs=0 len=4194304
2016-06-02 14:09:25.943698 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2611 10.7e0ff622 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [read 0~4194304] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e40a40c0 con 0x5640603ecb10
2016-06-02 14:09:25.943767 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:25.947865 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:25.947914 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=96993280 stripe_ofs=96993280 part_ofs=0 rule->part_size=0
2016-06-02 14:09:25.947948 7f77037d6700 20 rados->get_obj_iterate_cb oid=62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 obj-ofs=96993280 read_ofs=0 len=1885376
2016-06-02 14:09:25.947983 7f77037d6700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2612 10.ab707ff2 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 [read 0~1885376] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77e40a4cb0 con 0x5640603ecb10
2016-06-02 14:09:25.948141 7f77037d6700 20 rados->aio_operate r=0 bl.length=0
2016-06-02 14:09:25.948164 7f77037d6700 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1
2016-06-02 14:09:25.948174 7f77037d6700 20 RGWObjManifest::operator++(): result: ofs=98878656 stripe_ofs=98878656 part_ofs=0 rule->part_size=0
2016-06-02 14:09:25.952506 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 669 ==== osd_op_reply(2611 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_23 [read 0~4194304] v0'0 uv3170 ondisk = 0) v7 ==== 208+0+4194304 (940516275 0 1496770721) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:25.953452 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=92798976 len=4194304
2016-06-02 14:09:25.953529 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 670 ==== osd_op_reply(2612 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1__shadow_.cTw_0SM3yZSS-qi1b6KHCc0Ov7cN0wG_24 [read 0~1885376] v0'0 uv3175 ondisk = 0) v7 ==== 208+0+1885376 (3086244 0 2489803631) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:25.953568 7f78297fa700 20 get_obj_aio_completion_cb: io completion ofs=96993280 len=1885376
2016-06-02 14:09:26.183491 7f77037d6700  2 req 634:2.047965:s3:GET /lotr/atroll5:get_obj:completing
2016-06-02 14:09:26.183541 7f77037d6700  2 req 634:2.048015:s3:GET /lotr/atroll5:get_obj:op status=0
2016-06-02 14:09:26.183556 7f77037d6700  2 req 634:2.048030:s3:GET /lotr/atroll5:get_obj:http status=200
2016-06-02 14:09:26.183572 7f77037d6700  1 ====== req done req=0x7f77037d0790 op status=0 http_status=200 ======
2016-06-02 14:09:26.183852 7f77037d6700  1 civetweb: 0x7f77e4004060: 127.0.0.1 - - [02/Jun/2016:14:09:24 +0200] "GET /lotr/atroll5 HTTP/1.1" 200 0 - -
2016-06-02 14:09:26.547312 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:26.547347 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:26.547474 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:26 2016
2016-06-02 14:09:26.547477 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:26 2016
2016-06-02 14:09:26.547491 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:26.547491 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:26 2016
/admin/log/
2016-06-02 14:09:26.547532 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:0xZb7t6uRHTFgbofndnOQ6Cjr2c=
2016-06-02 14:09:26.547552 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:26.547558 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=339, easy_handle=0x7f773c380450
2016-06-02 14:09:26.547566 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:26.547574 7f77477fe700 20 link_request req_data=0x7f773c2d6c70 req_data->id=339, easy_handle=0x7f773c380450
2016-06-02 14:09:26.548313 7f77477fe700 10 receive_http_header
2016-06-02 14:09:26.548316 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:26.548318 7f77477fe700 10 receive_http_header
2016-06-02 14:09:26.548319 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001f4-00575021f6-3728-us-west
2016-06-02 14:09:26.548323 7f77477fe700 10 receive_http_header
2016-06-02 14:09:26.548323 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:26.548324 7f77477fe700 10 receive_http_header
2016-06-02 14:09:26.548325 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:26 GMT
2016-06-02 14:09:26.548326 7f77477fe700 10 receive_http_header
2016-06-02 14:09:26.548327 7f77477fe700 10 received header:
2016-06-02 14:09:26.548403 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:26.548759 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:26.548765 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:26.548768 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:26.548773 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:26.980965 7f7702fd5700 20 RGWEnv::set(): HTTP_HOST: localhost:8001
2016-06-02 14:09:26.980973 7f7702fd5700 20 RGWEnv::set(): HTTP_ACCEPT: */*
2016-06-02 14:09:26.980977 7f7702fd5700 20 RGWEnv::set(): HTTP_TRANSFER_ENCODING: chunked
2016-06-02 14:09:26.980978 7f7702fd5700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS 1555b35654ad1656d804:SN+L0ANvuwgtBFOC3/AYcVU5spk=
2016-06-02 14:09:26.980980 7f7702fd5700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:26 2016
2016-06-02 14:09:26.980980 7f7702fd5700 20 RGWEnv::set(): HTTP_EXPECT: 100-continue
2016-06-02 14:09:26.980981 7f7702fd5700 20 RGWEnv::set(): REQUEST_METHOD: POST
2016-06-02 14:09:26.980982 7f7702fd5700 20 RGWEnv::set(): REQUEST_URI: /admin/log
2016-06-02 14:09:26.980982 7f7702fd5700 20 RGWEnv::set(): QUERY_STRING: type=data&notify&source-zone=91441c60-0ca1-4118-9685-990e04272142&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:26.980983 7f7702fd5700 20 RGWEnv::set(): REMOTE_USER:
2016-06-02 14:09:26.980984 7f7702fd5700 20 RGWEnv::set(): SCRIPT_URI: /admin/log
2016-06-02 14:09:26.980985 7f7702fd5700 20 RGWEnv::set(): SERVER_PORT: 8001
2016-06-02 14:09:26.980985 7f7702fd5700 20 HTTP_ACCEPT=*/*
2016-06-02 14:09:26.980986 7f7702fd5700 20 HTTP_AUTHORIZATION=AWS 1555b35654ad1656d804:SN+L0ANvuwgtBFOC3/AYcVU5spk=
2016-06-02 14:09:26.980986 7f7702fd5700 20 HTTP_DATE=Thu Jun  2 12:09:26 2016
2016-06-02 14:09:26.980986 7f7702fd5700 20 HTTP_EXPECT=100-continue
2016-06-02 14:09:26.980987 7f7702fd5700 20 HTTP_HOST=localhost:8001
2016-06-02 14:09:26.980987 7f7702fd5700 20 HTTP_TRANSFER_ENCODING=chunked
2016-06-02 14:09:26.980988 7f7702fd5700 20 QUERY_STRING=type=data&notify&source-zone=91441c60-0ca1-4118-9685-990e04272142&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:26.980988 7f7702fd5700 20 REMOTE_USER=
2016-06-02 14:09:26.980989 7f7702fd5700 20 REQUEST_METHOD=POST
2016-06-02 14:09:26.980989 7f7702fd5700 20 REQUEST_URI=/admin/log
2016-06-02 14:09:26.980989 7f7702fd5700 20 SCRIPT_URI=/admin/log
2016-06-02 14:09:26.980990 7f7702fd5700 20 SERVER_PORT=8001
2016-06-02 14:09:26.980990 7f7702fd5700  1 ====== starting new request req=0x7f7702fcf790 =====
2016-06-02 14:09:26.981005 7f7702fd5700  2 req 636:0.000014::POST /admin/log::initializing for trans_id = tx00000000000000000027c-00575021f6-5e2e-us-east-1
2016-06-02 14:09:26.981007 7f7702fd5700 10 host=localhost
2016-06-02 14:09:26.981010 7f7702fd5700 20 subdomain= domain=localhost in_hosted_domain=1 in_hosted_domain_s3website=0
2016-06-02 14:09:26.981028 7f7702fd5700 10 handler=14RGWHandler_Log
2016-06-02 14:09:26.981029 7f7702fd5700  2 req 636:0.000038::POST /admin/log::getting op 4
2016-06-02 14:09:26.981033 7f7702fd5700 10 op=20RGWOp_DATALog_Notify
2016-06-02 14:09:26.981034 7f7702fd5700  2 req 636:0.000043::POST /admin/log:datalog_notify:authorizing
2016-06-02 14:09:26.981051 7f7702fd5700 10 get_canon_resource(): dest=/admin/log
2016-06-02 14:09:26.981053 7f7702fd5700 10 auth_hdr:
POST


Thu Jun  2 12:09:26 2016
/admin/log
2016-06-02 14:09:26.981078 7f7702fd5700 15 calculated digest=SN+L0ANvuwgtBFOC3/AYcVU5spk=
2016-06-02 14:09:26.981080 7f7702fd5700 15 auth_sign=SN+L0ANvuwgtBFOC3/AYcVU5spk=
2016-06-02 14:09:26.981080 7f7702fd5700 15 compare=0
2016-06-02 14:09:26.981082 7f7702fd5700 20 system request
2016-06-02 14:09:26.981084 7f7702fd5700  2 req 636:0.000093::POST /admin/log:datalog_notify:normalizing buckets and tenants
2016-06-02 14:09:26.981085 7f7702fd5700  2 req 636:0.000095::POST /admin/log:datalog_notify:init permissions
2016-06-02 14:09:26.981088 7f7702fd5700  2 req 636:0.000097::POST /admin/log:datalog_notify:recalculating target
2016-06-02 14:09:26.981089 7f7702fd5700  2 req 636:0.000098::POST /admin/log:datalog_notify:reading permissions
2016-06-02 14:09:26.981089 7f7702fd5700  2 req 636:0.000099::POST /admin/log:datalog_notify:init op
2016-06-02 14:09:26.981090 7f7702fd5700  2 req 636:0.000099::POST /admin/log:datalog_notify:verifying op mask
2016-06-02 14:09:26.981091 7f7702fd5700 20 required_mask= 0 user.op_mask=7
2016-06-02 14:09:26.981091 7f7702fd5700  2 req 636:0.000101::POST /admin/log:datalog_notify:verifying op permissions
2016-06-02 14:09:26.981093 7f7702fd5700  2 overriding permissions due to system operation
2016-06-02 14:09:26.981093 7f7702fd5700  2 req 636:0.000102::POST /admin/log:datalog_notify:verifying op params
2016-06-02 14:09:26.981094 7f7702fd5700  2 req 636:0.000103::POST /admin/log:datalog_notify:pre-executing
2016-06-02 14:09:26.981094 7f7702fd5700  2 req 636:0.000104::POST /admin/log:datalog_notify:executing
2016-06-02 14:09:27.949533 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:27.949637 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:27.949725 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:27 2016
2016-06-02 14:09:27.949728 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:27 2016
2016-06-02 14:09:27.949742 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:27.949742 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:27 2016
/admin/log/
2016-06-02 14:09:27.949782 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:n0Penrex10kDxzypNABqW6zGHCw=
2016-06-02 14:09:27.949799 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:27.949806 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=340, easy_handle=0x7f773c380450
2016-06-02 14:09:27.949813 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:27.949823 7f77477fe700 20 link_request req_data=0x7f773c18a6f0 req_data->id=340, easy_handle=0x7f773c380450
2016-06-02 14:09:27.950584 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.950587 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:27.950590 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.950590 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001f5-00575021f7-3728-us-west
2016-06-02 14:09:27.950594 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.950595 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:27.950597 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.950597 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:27 GMT
2016-06-02 14:09:27.950600 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.950600 7f77477fe700 10 received header:
2016-06-02 14:09:27.950652 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:27.950951 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:27.950957 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:27.950960 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:27.950965 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:27.982206 7f7702fd5700 20 execute(): read data: [{"key":36,"val":["lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1"]}]
2016-06-02 14:09:27.982274 7f7702fd5700 20 execute(): updated shard=36
2016-06-02 14:09:27.982276 7f7702fd5700 20 execute(): modified key=lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1
2016-06-02 14:09:27.982277 7f7702fd5700 20 wakeup_data_sync_shards: source_zone=91441c60-0ca1-4118-9685-990e04272142, shard_ids={36=lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1}
2016-06-02 14:09:27.982310 7f7702fd5700  2 req 636:1.001319::POST /admin/log:datalog_notify:completing
2016-06-02 14:09:27.982321 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:27.982328 7f7760ff9700 20 incremental_sync(): async update notification: lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1
2016-06-02 14:09:27.982351 7f7702fd5700  2 req 636:1.001361::POST /admin/log:datalog_notify:op status=0
2016-06-02 14:09:27.982354 7f7702fd5700  2 req 636:1.001364::POST /admin/log:datalog_notify:http status=200
2016-06-02 14:09:27.982356 7f7702fd5700  1 ====== req done req=0x7f7702fcf790 op status=0 http_status=200 ======
2016-06-02 14:09:27.982510 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c2e2180:24RGWDataSyncSingleEntryCR: operate()
2016-06-02 14:09:27.982510 7f7702fd5700  1 civetweb: 0x7f77e0003d10: 127.0.0.1 - - [02/Jun/2016:14:09:26 +0200] "POST /admin/log HTTP/1.1" 200 0 - -
2016-06-02 14:09:27.982521 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:27.982526 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:27.982532 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:27.982629 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:27 2016
2016-06-02 14:09:27.982633 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:27 2016
2016-06-02 14:09:27.982647 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:27.982647 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:27 2016
/admin/log/
2016-06-02 14:09:27.982692 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:n0Penrex10kDxzypNABqW6zGHCw=
2016-06-02 14:09:27.982713 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:27.982721 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=341, easy_handle=0x7f773c380450
2016-06-02 14:09:27.982729 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:27.982732 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:32RGWReadBucketSyncStatusCoroutine: operate()
2016-06-02 14:09:27.982745 7f77477fe700 20 link_request req_data=0x7f773c066bf0 req_data->id=341, easy_handle=0x7f773c380450
2016-06-02 14:09:27.982747 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c671720:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:27.982753 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c671720:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:27.982768 7f7760ff9700 20 enqueued request req=0x7f773c2c5ab0
2016-06-02 14:09:27.982770 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:27.982771 7f7760ff9700 20 req: 0x7f773c2c5ab0
2016-06-02 14:09:27.982774 7f7760ff9700 20 run: stack=0x7f773c0ac590 is io blocked
2016-06-02 14:09:27.982807 7f77ef7fe700 20 dequeued request req=0x7f773c2c5ab0
2016-06-02 14:09:27.982812 7f77ef7fe700 20 RGWWQ: empty
2016-06-02 14:09:27.982825 7f77ef7fe700 10 cache get: name=us-east-1.rgw.log+bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=3, cached=11)
2016-06-02 14:09:27.982877 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c671720:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:27.982880 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c671720:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:27.982882 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c671720:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:27.982883 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c671720:25RGWSimpleRadosReadAttrsCR: operate()
2016-06-02 14:09:27.983229 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:32RGWReadBucketSyncStatusCoroutine: operate()
2016-06-02 14:09:27.983240 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:27.983242 7f7760ff9700 20 operate(): sync status for bucket lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1: 2
2016-06-02 14:09:27.983250 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:27.983253 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:27.983262 7f7760ff9700 20 enqueued request req=0x7f773c0a0ca0
2016-06-02 14:09:27.983264 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:27.983264 7f7760ff9700 20 req: 0x7f773c0a0ca0
2016-06-02 14:09:27.983267 7f7760ff9700 20 run: stack=0x7f773c0ac590 is io blocked
2016-06-02 14:09:27.983273 7f77a97fa700 20 dequeued request req=0x7f773c0a0ca0
2016-06-02 14:09:27.983277 7f77a97fa700 20 RGWWQ: empty
2016-06-02 14:09:27.983284 7f77a97fa700 20 reading from us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:27.983294 7f77a97fa700 20 get_system_obj_state: rctx=0x7f77a97f8560 obj=us-east-1.rgw.data.root:.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 state=0x7f7768018d48 s->prefetch_data=0
2016-06-02 14:09:27.983299 7f77a97fa700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=22, cached=23)
2016-06-02 14:09:27.983305 7f77a97fa700 20 get_system_obj_state: s->obj_tag was set empty
2016-06-02 14:09:27.983307 7f77a97fa700 10 cache get: name=us-east-1.rgw.data.root+.bucket.meta.lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 : hit (requested=17, cached=23)
2016-06-02 14:09:27.983443 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:27.983448 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:27.983449 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:27.983450 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:26RGWGetBucketInstanceInfoCR: operate()
2016-06-02 14:09:27.983781 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:27.983783 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.983785 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:27.983785 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:27.983787 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:27.983787 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.983787 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001f6-00575021f7-3728-us-west
2016-06-02 14:09:27.983790 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.983791 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:27.983792 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.983793 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:27 GMT
2016-06-02 14:09:27.983794 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.983794 7f77477fe700 10 received header:
2016-06-02 14:09:27.983796 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:27.983821 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c0d79d0:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:27.983828 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:27.983830 7f7760ff9700 20 run: stack=0x7f773c0ac590 is_blocked_by_stack()=0 is_sleeping=1 waiting_for_child()=0
2016-06-02 14:09:27.983831 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c3335e0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:27.983833 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c3335e0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:27.983842 7f7760ff9700 20 enqueued request req=0x7f773c1a4660
2016-06-02 14:09:27.983844 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:27.983844 7f7760ff9700 20 req: 0x7f773c1a4660
2016-06-02 14:09:27.983847 7f7760ff9700 20 run: stack=0x7f773c1a8740 is io blocked
2016-06-02 14:09:27.983850 7f77effff700 20 dequeued request req=0x7f773c1a4660
2016-06-02 14:09:27.983852 7f77effff700 20 RGWWQ: empty
2016-06-02 14:09:27.983881 7f77effff700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2613 5.944771f8 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77dc010d40 con 0x5640603f1eb0
2016-06-02 14:09:27.994624 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 896 ==== osd_op_reply(2613 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call] v32'2381 uv2381 ondisk = 0) v7 ==== 225+0+0 (106038407 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:27.994734 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c3335e0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:27.994753 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c3335e0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:27.994754 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c3335e0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:27.994756 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c3335e0:20RGWSimpleRadosLockCR: operate()
2016-06-02 14:09:27.994985 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c0d79d0:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:27.995003 7f7760ff9700 20 run: stack=0x7f773c1a8740 is io blocked
2016-06-02 14:09:27.995005 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:27.995006 7f7760ff9700 20 abhi: starting a new RGWBucketIncSyncShardMarkerTrack with marker oidbucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1
2016-06-02 14:09:27.995007 7f7760ff9700 20 operate(): listing bilog for incremental sync
2016-06-02 14:09:27.995014 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c5d7ae0:23RGWListBucketIndexLogCR: operate()
2016-06-02 14:09:27.995020 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:27.995021 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:27.995109 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:27 2016
2016-06-02 14:09:27.995111 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:27 2016
2016-06-02 14:09:27.995124 7f7760ff9700 10 get_canon_resource(): dest=/admin/log
2016-06-02 14:09:27.995125 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:27 2016
/admin/log
2016-06-02 14:09:27.995147 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:E0H5UaCSviApZMrm27R26ddw0rw=
2016-06-02 14:09:27.995165 7f7760ff9700 20 sending request to http://localhost:8002/admin/log?bucket-instance=lotr%3A62bc922d-f295-4067-ae36-e23e2f231aad.24099.1&format=json&marker=00000000472.1089.3&type=bucket-index&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:27.995179 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=342, easy_handle=0x7f773c665a80
2016-06-02 14:09:27.995185 7f7760ff9700 20 run: stack=0x7f773c0ac590 is io blocked
2016-06-02 14:09:27.995190 7f77477fe700 20 link_request req_data=0x7f773c12cc60 req_data->id=342, easy_handle=0x7f773c665a80
2016-06-02 14:09:27.996043 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.996048 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:27.996051 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.996051 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001f7-00575021f7-3728-us-west
2016-06-02 14:09:27.996065 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.996066 7f77477fe700 10 received header:Content-Length: 577
2016-06-02 14:09:27.996067 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.996067 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:27 GMT
2016-06-02 14:09:27.996069 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.996069 7f77477fe700 10 received header:
2016-06-02 14:09:27.996146 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:27.996337 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:27.996341 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:27.996342 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:27.996346 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c5d7ae0:23RGWListBucketIndexLogCR: operate()
2016-06-02 14:09:27.996348 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:27.996351 7f7760ff9700 20 parsed entry: id=00000000473.1090.2 iter->object=atroll5 iter->instance= name=atroll5 instance= ns=
2016-06-02 14:09:27.996355 7f7760ff9700 20 [inc sync] skipping object: lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1/atroll5: non-complete operation
2016-06-02 14:09:27.996357 7f7760ff9700 20 parsed entry: id=00000000474.1091.3 iter->object=atroll5 iter->instance= name=atroll5 instance= ns=
2016-06-02 14:09:27.996359 7f7760ff9700 20 [inc sync] syncing object: lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1:-1/atroll5
2016-06-02 14:09:27.996360 7f7760ff9700 20 abhi: couldn't find key, no sync in progress for object atroll5
2016-06-02 14:09:27.996365 7f7760ff9700 20 operate(): entry->timestamp=2016-06-02 14:09:19.0.256194s
2016-06-02 14:09:27.996395 7f7760ff9700 20 bucket sync single entry (source_zone=91441c60-0ca1-4118-9685-990e04272142) b=lotr(@{i=us-east-1.rgw.buckets.index,e=us-east-1.rgw.buckets.non-ec}us-east-1.rgw.buckets.data[62bc922d-f295-4067-ae36-e23e2f231aad.24099.1]):-1/atroll5[0] log_entry=00000000474.1091.3 op=0 op_state=1
2016-06-02 14:09:27.996401 7f7760ff9700 20 operate(): listing bilog for incremental sync
2016-06-02 14:09:27.996408 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773caf0370:26RGWBucketSyncSingleEntryCRINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE11rgw_obj_keyE: operate()
2016-06-02 14:09:27.996412 7f7760ff9700  5 bucket sync: sync obj: 91441c60-0ca1-4118-9685-990e04272142/lotr(@{i=us-east-1.rgw.buckets.index,e=us-east-1.rgw.buckets.non-ec}us-east-1.rgw.buckets.data[62bc922d-f295-4067-ae36-e23e2f231aad.24099.1])/atroll5[0]
2016-06-02 14:09:27.996422 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c5d7ae0:23RGWListBucketIndexLogCR: operate()
2016-06-02 14:09:27.996429 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773c0c0780:19RGWFetchRemoteObjCR: operate()
2016-06-02 14:09:27.996431 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:27.996433 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773c0c0780:19RGWFetchRemoteObjCR: operate()
2016-06-02 14:09:27.996445 7f7760ff9700 20 enqueued request req=0x7f773c1656e0
2016-06-02 14:09:27.996446 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:27.996447 7f7760ff9700 20 req: 0x7f773c1656e0
2016-06-02 14:09:27.996450 7f7760ff9700 20 run: stack=0x7f773c17e830 is io blocked
2016-06-02 14:09:27.996452 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:27.996453 7f77eeffd700 20 dequeued request req=0x7f773c1656e0
2016-06-02 14:09:27.996455 7f77eeffd700 20 RGWWQ: empty
2016-06-02 14:09:27.996504 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:27 2016
2016-06-02 14:09:27.996507 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:27 2016
2016-06-02 14:09:27.996507 7f77eeffd700 20 get_obj_state: rctx=0x7f77eeffb190 obj=lotr:atroll5 state=0x7f77d400d5d8 s->prefetch_data=0
2016-06-02 14:09:27.996510 7f7760ff9700 10 get_canon_resource(): dest=/admin/log
2016-06-02 14:09:27.996511 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:27 2016
/admin/log
2016-06-02 14:09:27.996532 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:E0H5UaCSviApZMrm27R26ddw0rw=
2016-06-02 14:09:27.996530 7f77eeffd700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2614 10.a8f56d53 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1_atroll5 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f77d401d5c0 con 0x5640603f1eb0
2016-06-02 14:09:27.996540 7f7760ff9700 20 sending request to http://localhost:8002/admin/log?bucket-instance=lotr%3A62bc922d-f295-4067-ae36-e23e2f231aad.24099.1&format=json&marker=00000000474.1091.3&type=bucket-index&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:27.996545 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=343, easy_handle=0x7f773c665a80
2016-06-02 14:09:27.996551 7f7760ff9700 20 run: stack=0x7f773c0ac590 is io blocked
2016-06-02 14:09:27.996556 7f77477fe700 20 link_request req_data=0x7f773c17d670 req_data->id=343, easy_handle=0x7f773c665a80
2016-06-02 14:09:27.996816 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 897 ==== osd_op_reply(2614 62bc922d-f295-4067-ae36-e23e2f231aad.24099.1_atroll5 [getxattrs,stat] v0'0 uv3109 ondisk = 0) v7 ==== 214+0+996 (3948354515 0 4250806884) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:27.996851 7f77eeffd700 10 manifest: total_size = 98878656
2016-06-02 14:09:27.996856 7f77eeffd700 20 get_obj_state: setting s->obj_tag to 91441c60-0ca1-4118-9685-990e04272142.14120.350
2016-06-02 14:09:27.996929 7f77eeffd700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:27 2016
2016-06-02 14:09:27.996931 7f77eeffd700 20 RGWEnv::set(): HTTP_DEST_PG_VER: 363
2016-06-02 14:09:27.996932 7f77eeffd700 20 RGWEnv::set(): HTTP_DEST_ZONE_SHORT_ID: 3526696609
2016-06-02 14:09:27.996933 7f77eeffd700 20 RGWEnv::set(): HTTP_IF_MODIFIED_SINCE: 2016-06-02 12:09:19.256194244Z
2016-06-02 14:09:27.996935 7f77eeffd700 20 > HTTP_DATE -> Thu Jun  2 12:09:27 2016
2016-06-02 14:09:27.996935 7f77eeffd700 20 > HTTP_DEST_PG_VER -> 363
2016-06-02 14:09:27.996938 7f77eeffd700 20 > HTTP_DEST_ZONE_SHORT_ID -> 3526696609
2016-06-02 14:09:27.996939 7f77eeffd700 20 > HTTP_IF_MODIFIED_SINCE -> 2016-06-02 12:09:19.256194244Z
2016-06-02 14:09:27.996943 7f77eeffd700 10 get_canon_resource(): dest=/lotr/atroll5
2016-06-02 14:09:27.996943 7f77eeffd700 10 generated canonical header: GET


Thu Jun  2 12:09:27 2016
/lotr/atroll5
2016-06-02 14:09:27.996966 7f77eeffd700 15 generated auth header: AWS 1555b35654ad1656d804:kYtHMcQoWi3voSCT+KjcEg01ejY=
2016-06-02 14:09:27.996974 7f77eeffd700 20 sending request to http://localhost:8002/lotr/atroll5?rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a&rgwx-prepend-metadata=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:27.996981 7f77eeffd700 20 register_request mgr=0x7f77d401e0b0 req_data->id=0, easy_handle=0x7f77d4035ca0
2016-06-02 14:09:27.996987 7f77eeffd700 20 link_request req_data=0x7f77d4035a80 req_data->id=0, easy_handle=0x7f77d4035ca0
2016-06-02 14:09:27.997183 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.997187 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:27.997188 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.997188 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001f8-00575021f7-3728-us-west
2016-06-02 14:09:27.997191 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.997191 7f77477fe700 10 received header:Content-Length: 2
2016-06-02 14:09:27.997193 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.997193 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:27 GMT
2016-06-02 14:09:27.997194 7f77477fe700 10 receive_http_header
2016-06-02 14:09:27.997195 7f77477fe700 10 received header:
2016-06-02 14:09:28.000288 7f77eeffd700 10 receive_http_header
2016-06-02 14:09:28.000291 7f77eeffd700 10 received header:HTTP/1.1 100 CONTINUE
2016-06-02 14:09:28.000292 7f77eeffd700 10 received header:HTTP/1.1
2016-06-02 14:09:28.000293 7f77eeffd700 10 receive_http_header
2016-06-02 14:09:28.000294 7f77eeffd700 10 received header:
2016-06-02 14:09:28.000354 7f77eeffd700 10 receive_http_header
2016-06-02 14:09:28.000357 7f77eeffd700 10 received header:HTTP/1.1 304 Not Modified
2016-06-02 14:09:28.000358 7f77eeffd700 10 receive_http_header
2016-06-02 14:09:28.000358 7f77eeffd700 10 received header:x-amz-request-id: tx0000000000000000001f9-00575021f7-3728-us-west
2016-06-02 14:09:28.000361 7f77eeffd700 10 receive_http_header
2016-06-02 14:09:28.000361 7f77eeffd700 10 received header:Content-Type: binary/octet-stream
2016-06-02 14:09:28.000363 7f77eeffd700 10 receive_http_header
2016-06-02 14:09:28.000363 7f77eeffd700 10 received header:Date: Thu, 02 Jun 2016 12:09:27 GMT
2016-06-02 14:09:28.000368 7f77eeffd700 10 receive_http_header
2016-06-02 14:09:28.000368 7f77eeffd700 10 received header:
2016-06-02 14:09:28.000592 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773c0c0780:19RGWFetchRemoteObjCR: operate()
2016-06-02 14:09:28.000607 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773c0c0780:19RGWFetchRemoteObjCR: operate()
2016-06-02 14:09:28.000609 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773c0c0780:19RGWFetchRemoteObjCR: operate()
2016-06-02 14:09:28.000611 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773c0c0780:19RGWFetchRemoteObjCR: operate()
2016-06-02 14:09:28.000854 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773caf0370:26RGWBucketSyncSingleEntryCRINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE11rgw_obj_keyE: operate()
2016-06-02 14:09:28.000861 7f7760ff9700 20 abhi: deleting markeratroll5marker: 00000000474.1091.3
2016-06-02 14:09:28.000870 7f7760ff9700 20 store_marker(): updating marker marker_oid=bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 marker=00000000474.1091.3
2016-06-02 14:09:28.000878 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773c028a90:26RGWSimpleRadosWriteAttrsCR: operate()
2016-06-02 14:09:28.000880 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773c028a90:26RGWSimpleRadosWriteAttrsCR: operate()
2016-06-02 14:09:28.000895 7f7760ff9700 20 enqueued request req=0x7f773c0af910
2016-06-02 14:09:28.000897 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:28.000898 7f7760ff9700 20 req: 0x7f773c0af910
2016-06-02 14:09:28.000901 7f7760ff9700 20 run: stack=0x7f773c17e830 is io blocked
2016-06-02 14:09:28.000920 7f77a8ff9700 20 dequeued request req=0x7f773c0af910
2016-06-02 14:09:28.000924 7f77a8ff9700 20 RGWWQ: empty
2016-06-02 14:09:28.000947 7f77a8ff9700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2615 5.944771f8 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [setxattr inc_marker (28)] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f775c018250 con 0x5640603f1eb0
2016-06-02 14:09:28.006984 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 898 ==== osd_op_reply(2615 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [setxattr (28)] v32'2382 uv2382 ondisk = 0) v7 ==== 225+0+0 (4220557371 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:28.007010 7f77a8ff9700 10 cache put: name=us-east-1.rgw.log+bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 info.flags=8
2016-06-02 14:09:28.007017 7f77a8ff9700 10 moving us-east-1.rgw.log+bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 to cache LRU end
2016-06-02 14:09:28.007018 7f77a8ff9700 10 appending xattr: name=inc_marker bl.length()=28
2016-06-02 14:09:28.007029 7f77a8ff9700 10 distributing notification oid=notify.6 bl.length()=375
2016-06-02 14:09:28.007049 7f77a8ff9700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2616 2.97c520d4 notify.6 [notify cookie 140150621411808] snapc 0=[] ondisk+read+known_if_redirected e32) v7 -- ?+0 0x7f775c003bf0 con 0x5640603ecb10
2016-06-02 14:09:28.007259 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 671 ==== watch-notify(notify (1) cookie 94834492718368 notify 137438953608 ret 0) v3 ==== 417+0+0 (1473324886 0 0) 0x7f780401b2e0 con 0x5640603ecb10
2016-06-02 14:09:28.007277 7f78297fa700 10 RGWWatcher::handle_notify()  notify_id 137438953608 cookie 94834492718368 notifier 24110 bl.length()=375
2016-06-02 14:09:28.007290 7f78297fa700 10 cache put: name=us-east-1.rgw.log+bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 info.flags=8
2016-06-02 14:09:28.007293 7f78297fa700 10 moving us-east-1.rgw.log+bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 to cache LRU end
2016-06-02 14:09:28.007294 7f78297fa700 10 appending xattr: name=inc_marker bl.length()=28
2016-06-02 14:09:28.007309 7f78297fa700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2617 2.97c520d4 notify.6 [notify-ack 0~0] snapc 0=[] read+known_if_redirected e32) v7 -- ?+0 0x7f77e000a7a0 con 0x5640603ecb10
2016-06-02 14:09:28.007340 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 672 ==== osd_op_reply(2616 notify.6 [notify cookie 140150621411808] v0'0 uv1 ondisk = 0) v7 ==== 128+0+8 (2361979270 0 2703197749) 0x7f780401a280 con 0x5640603ecb10
2016-06-02 14:09:28.007612 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 673 ==== watch-notify(notify_complete (2) cookie 140150621411808 notify 137438953608 ret 0) v3 ==== 42+0+28 (4258368746 0 174158344) 0x7f780401b2e0 con 0x5640603ecb10
2016-06-02 14:09:28.007643 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 674 ==== osd_op_reply(2617 notify.6 [notify-ack 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3357944540 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:28.007660 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773c028a90:26RGWSimpleRadosWriteAttrsCR: operate()
2016-06-02 14:09:28.007669 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773c028a90:26RGWSimpleRadosWriteAttrsCR: operate()
2016-06-02 14:09:28.007671 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773c028a90:26RGWSimpleRadosWriteAttrsCR: operate()
2016-06-02 14:09:28.007672 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773c028a90:26RGWSimpleRadosWriteAttrsCR: operate()
2016-06-02 14:09:28.007894 7f7760ff9700 20 cr:s=0x7f773c17e830:op=0x7f773caf0370:26RGWBucketSyncSingleEntryCRINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE11rgw_obj_keyE: operate()
2016-06-02 14:09:28.007898 7f7760ff9700 20 run: stack=0x7f773c17e830 is done
2016-06-02 14:09:28.021380 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:28.021506 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:28.021509 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:28.021513 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:28.021519 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:28.033369 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:28.033663 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:28.033667 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:28.033668 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c0a1f90:21RGWReadRESTResourceCRINSt7__cxx114listI16rgw_bi_log_entrySaIS2_EEEE: operate()
2016-06-02 14:09:28.033682 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c5d7ae0:23RGWListBucketIndexLogCR: operate()
2016-06-02 14:09:28.033685 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:28.033687 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:28.033691 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:28.033692 7f7760ff9700 20 collect(): s=0x7f773c0ac590 stack=0x7f773c1a8740 is still running
2016-06-02 14:09:28.033693 7f7760ff9700 20 collect(): s=0x7f773c0ac590 stack=0x7f773c17e830 is complete
2016-06-02 14:09:28.033694 7f7760ff9700 20 run: stack=0x7f773c0ac590 is_blocked_by_stack()=0 is_sleeping=0 waiting_for_child()=1
2016-06-02 14:09:28.033695 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c0d79d0:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:28.033767 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c3335e0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:28.033768 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c3335e0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:28.033779 7f7760ff9700 20 enqueued request req=0x7f773c3186c0
2016-06-02 14:09:28.033780 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:28.033780 7f7760ff9700 20 req: 0x7f773c3186c0
2016-06-02 14:09:28.033783 7f7760ff9700 20 run: stack=0x7f773c1a8740 is io blocked
2016-06-02 14:09:28.033787 7f77c77fe700 20 dequeued request req=0x7f773c3186c0
2016-06-02 14:09:28.033790 7f77c77fe700 20 RGWWQ: empty
2016-06-02 14:09:28.033839 7f77c77fe700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2618 5.944771f8 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77a401a540 con 0x5640603f1eb0
2016-06-02 14:09:28.041679 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 899 ==== osd_op_reply(2618 bucket.sync-status.91441c60-0ca1-4118-9685-990e04272142:lotr:62bc922d-f295-4067-ae36-e23e2f231aad.24099.1 [call] v32'2383 uv2383 ondisk = 0) v7 ==== 225+0+0 (2093452865 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:28.041753 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c3335e0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:28.041760 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c3335e0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:28.041772 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c3335e0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:28.041773 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c3335e0:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:28.041999 7f7760ff9700 20 cr:s=0x7f773c1a8740:op=0x7f773c0d79d0:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:28.042002 7f7760ff9700 20 run: stack=0x7f773c1a8740 is done
2016-06-02 14:09:28.042005 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:28.042005 7f7760ff9700 20 collect(): s=0x7f773c0ac590 stack=0x7f773c1a8740 is complete
2016-06-02 14:09:28.042007 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c136d60:31RGWBucketShardIncrementalSyncCR: operate()
2016-06-02 14:09:28.042012 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c30a290:25RGWRunBucketSyncCoroutine: operate()
2016-06-02 14:09:28.042017 7f7760ff9700 20 cr:s=0x7f773c0ac590:op=0x7f773c2e2180:24RGWDataSyncSingleEntryCR: operate()
2016-06-02 14:09:28.042021 7f7760ff9700 20 run: stack=0x7f773c0ac590 is done
2016-06-02 14:09:28.951753 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:28.951799 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:28.951927 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:28 2016
2016-06-02 14:09:28.951931 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:28 2016
2016-06-02 14:09:28.951945 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:28.951946 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:28 2016
/admin/log/
2016-06-02 14:09:28.951981 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:KRNRw0hIjCHWEFrxvHejShvq5eM=
2016-06-02 14:09:28.952011 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:28.952018 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=344, easy_handle=0x7f773c380450
2016-06-02 14:09:28.952026 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:28.952063 7f77477fe700 20 link_request req_data=0x7f773c2d28d0 req_data->id=344, easy_handle=0x7f773c380450
2016-06-02 14:09:28.953084 7f77477fe700 10 receive_http_header
2016-06-02 14:09:28.953088 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:28.953090 7f77477fe700 10 receive_http_header
2016-06-02 14:09:28.953090 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001fa-00575021f8-3728-us-west
2016-06-02 14:09:28.953103 7f77477fe700 10 receive_http_header
2016-06-02 14:09:28.953104 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:28.953105 7f77477fe700 10 receive_http_header
2016-06-02 14:09:28.953106 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:28 GMT
2016-06-02 14:09:28.953107 7f77477fe700 10 receive_http_header
2016-06-02 14:09:28.953107 7f77477fe700 10 received header:
2016-06-02 14:09:28.953160 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:28.953564 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:28.953568 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:28.953582 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:28.953589 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:29.396981 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2619 2.4322fa9f notify.0 [watch ping cookie 94834492702224] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f8015850 con 0x5640603ecb10
2016-06-02 14:09:29.397009 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2620 2.88aa5c95 notify.2 [watch ping cookie 94834492706960] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f8017460 con 0x5640603ecb10
2016-06-02 14:09:29.397026 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2621 2.f8c99aee notify.3 [watch ping cookie 94834492709872] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f8015ee0 con 0x5640603ecb10
2016-06-02 14:09:29.397095 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2622 2.a204812d notify.4 [watch ping cookie 94834492712752] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f8019410 con 0x5640603ecb10
2016-06-02 14:09:29.397104 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2623 2.31099063 notify.5 [watch ping cookie 94834492715648] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f8018880 con 0x5640603ecb10
2016-06-02 14:09:29.397114 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2624 2.97c520d4 notify.6 [watch ping cookie 94834492718368] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f801af90 con 0x5640603ecb10
2016-06-02 14:09:29.397121 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2625 2.16dafda0 notify.1 [watch ping cookie 94834492703856] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f801b870 con 0x5640603f1eb0
2016-06-02 14:09:29.397130 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2626 2.84ada7c9 notify.7 [watch ping cookie 94834492721344] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f801be80 con 0x5640603f1eb0
2016-06-02 14:09:29.397162 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- ping magic: 0 v1 -- ?+0 0x7f77f8017460 con 0x5640603ecb10
2016-06-02 14:09:29.397165 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- ping magic: 0 v1 -- ?+0 0x7f77f8015ee0 con 0x5640603f1eb0
2016-06-02 14:09:29.397446 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 900 ==== osd_op_reply(2625 notify.1 [watch ping cookie 94834492703856] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (689729908 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:29.397565 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 901 ==== osd_op_reply(2626 notify.7 [watch ping cookie 94834492721344] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1842238908 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:29.397618 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 675 ==== osd_op_reply(2620 notify.2 [watch ping cookie 94834492706960] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3592388915 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:29.397660 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 676 ==== osd_op_reply(2622 notify.4 [watch ping cookie 94834492712752] v0'0 uv3 ondisk = 0) v7 ==== 128+0+0 (1823463381 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:29.397700 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 677 ==== osd_op_reply(2619 notify.0 [watch ping cookie 94834492702224] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (4158488181 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:29.397739 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 678 ==== osd_op_reply(2621 notify.3 [watch ping cookie 94834492709872] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1695028518 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:29.397839 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 679 ==== osd_op_reply(2623 notify.5 [watch ping cookie 94834492715648] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3716607297 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:29.397890 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 680 ==== osd_op_reply(2624 notify.6 [watch ping cookie 94834492718368] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1447217987 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:29.502103 7f76e6f9d700  1 -- 127.0.0.1:0/2056988180 --> 127.0.0.1:6808/30507 -- osd_op(client.24113.0:11 1.fd9d5d69 realms.00b17c78-da61-464b-906a-81dbe28b899d.control [watch ping cookie 94834493713904] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77fc026620 con 0x5640604f3290
2016-06-02 14:09:29.502137 7f76e6f9d700  1 -- 127.0.0.1:0/2056988180 --> 127.0.0.1:6808/30507 -- ping magic: 0 v1 -- ?+0 0x7f77fc026c30 con 0x5640604f3290
2016-06-02 14:09:29.502663 7f78281f2700  1 -- 127.0.0.1:0/2056988180 <== osd.2 127.0.0.1:6808/30507 11 ==== osd_op_reply(11 realms.00b17c78-da61-464b-906a-81dbe28b899d.control [watch ping cookie 94834493713904] v0'0 uv1 ondisk = 0) v7 ==== 171+0+0 (3084123279 0 0) 0x7f77e801def0 con 0x5640604f3290
2016-06-02 14:09:29.954022 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:29.954151 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:29.954270 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:29 2016
2016-06-02 14:09:29.954273 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:29 2016
2016-06-02 14:09:29.954287 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:29.954288 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:29 2016
/admin/log/
2016-06-02 14:09:29.954322 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:fkmpzoqm5IotUmgjIvNkW7l3XGs=
2016-06-02 14:09:29.954340 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:29.954357 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=345, easy_handle=0x7f773c380450
2016-06-02 14:09:29.954365 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:29.954390 7f77477fe700 20 link_request req_data=0x7f773c045400 req_data->id=345, easy_handle=0x7f773c380450
2016-06-02 14:09:29.955318 7f77477fe700 10 receive_http_header
2016-06-02 14:09:29.955323 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:29.955326 7f77477fe700 10 receive_http_header
2016-06-02 14:09:29.955327 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001fb-00575021f9-3728-us-west
2016-06-02 14:09:29.955341 7f77477fe700 10 receive_http_header
2016-06-02 14:09:29.955341 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:29.955342 7f77477fe700 10 receive_http_header
2016-06-02 14:09:29.955343 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:29 GMT
2016-06-02 14:09:29.955344 7f77477fe700 10 receive_http_header
2016-06-02 14:09:29.955345 7f77477fe700 10 received header:
2016-06-02 14:09:29.955427 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:29.955789 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:29.955793 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:29.955795 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:29.955802 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:30.955999 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:30.956132 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:30.956233 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:30 2016
2016-06-02 14:09:30.956236 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:30 2016
2016-06-02 14:09:30.956250 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:30.956250 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:30 2016
/admin/log/
2016-06-02 14:09:30.956281 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:9eYAVwSVUWuclMfhJ84BtTbpKNo=
2016-06-02 14:09:30.956300 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:30.956316 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=346, easy_handle=0x7f773c380450
2016-06-02 14:09:30.956323 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:30.956348 7f77477fe700 20 link_request req_data=0x7f773c073bd0 req_data->id=346, easy_handle=0x7f773c380450
2016-06-02 14:09:30.957186 7f77477fe700 10 receive_http_header
2016-06-02 14:09:30.957189 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:30.957191 7f77477fe700 10 receive_http_header
2016-06-02 14:09:30.957191 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001fc-00575021fa-3728-us-west
2016-06-02 14:09:30.957195 7f77477fe700 10 receive_http_header
2016-06-02 14:09:30.957196 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:30.957197 7f77477fe700 10 receive_http_header
2016-06-02 14:09:30.957198 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:30 GMT
2016-06-02 14:09:30.957199 7f77477fe700 10 receive_http_header
2016-06-02 14:09:30.957200 7f77477fe700 10 received header:
2016-06-02 14:09:30.957253 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:30.957589 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:30.957595 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:30.957597 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:30.957603 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:32.158335 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:32.158449 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:32.158543 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:32 2016
2016-06-02 14:09:32.158546 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:32 2016
2016-06-02 14:09:32.158560 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:32.158560 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:32 2016
/admin/log/
2016-06-02 14:09:32.158589 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:86n2vZd+yd2kIJFbijCsLag+o0A=
2016-06-02 14:09:32.158607 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:32.158623 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=347, easy_handle=0x7f773c380450
2016-06-02 14:09:32.158630 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:32.158639 7f77477fe700 20 link_request req_data=0x7f773c080980 req_data->id=347, easy_handle=0x7f773c380450
2016-06-02 14:09:32.159514 7f77477fe700 10 receive_http_header
2016-06-02 14:09:32.159517 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:32.159519 7f77477fe700 10 receive_http_header
2016-06-02 14:09:32.159519 7f77477fe700 10 received header:x-amz-request-id: tx0000000000000000001fd-00575021fc-3728-us-west
2016-06-02 14:09:32.159522 7f77477fe700 10 receive_http_header
2016-06-02 14:09:32.159522 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:32.159523 7f77477fe700 10 receive_http_header
2016-06-02 14:09:32.159524 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:32 GMT
2016-06-02 14:09:32.159525 7f77477fe700 10 receive_http_header
2016-06-02 14:09:32.159525 7f77477fe700 10 received header:
2016-06-02 14:09:32.159575 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:32.159877 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:32.159881 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:32.159883 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:32.159888 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:33.160838 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:33.160956 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:33.161059 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:33 2016
2016-06-02 14:09:33.161062 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:33 2016
2016-06-02 14:09:33.161077 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:33.161077 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:33 2016
/admin/log/
2016-06-02 14:09:33.161109 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:8Vow7N++Wnaq+n0OS3hT4IX5Uoo=
2016-06-02 14:09:33.161127 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:33.161143 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=348, easy_handle=0x7f773c380450
2016-06-02 14:09:33.161150 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:33.161157 7f77477fe700 20 link_request req_data=0x7f773c5ca130 req_data->id=348, easy_handle=0x7f773c380450
2016-06-02 14:09:33.162135 7f77477fe700 10 receive_http_header
2016-06-02 14:09:33.162138 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:33.162140 7f77477fe700 10 receive_http_header
2016-06-02 14:09:33.162141 7f77477fe700 10 received header:x-amz-request-id: tx000000000000000000201-00575021fd-3728-us-west
2016-06-02 14:09:33.162154 7f77477fe700 10 receive_http_header
2016-06-02 14:09:33.162154 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:33.162156 7f77477fe700 10 receive_http_header
2016-06-02 14:09:33.162156 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:33 GMT
2016-06-02 14:09:33.162158 7f77477fe700 10 receive_http_header
2016-06-02 14:09:33.162159 7f77477fe700 10 received header:
2016-06-02 14:09:33.162244 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:33.162562 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:33.162567 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:33.162569 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:33.162574 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:34.112751 7f770a7e4700 20 RGWEnv::set(): HTTP_HOST: localhost:8001
2016-06-02 14:09:34.112758 7f770a7e4700 20 RGWEnv::set(): HTTP_ACCEPT: */*
2016-06-02 14:09:34.112760 7f770a7e4700 20 RGWEnv::set(): HTTP_TRANSFER_ENCODING: chunked
2016-06-02 14:09:34.112762 7f770a7e4700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS 1555b35654ad1656d804:F45z4l4bjPDDsSGmNZku0jBvHug=
2016-06-02 14:09:34.112773 7f770a7e4700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:34 2016
2016-06-02 14:09:34.112774 7f770a7e4700 20 RGWEnv::set(): HTTP_EXPECT: 100-continue
2016-06-02 14:09:34.112775 7f770a7e4700 20 RGWEnv::set(): REQUEST_METHOD: GET
2016-06-02 14:09:34.112776 7f770a7e4700 20 RGWEnv::set(): REQUEST_URI: /admin/log/
2016-06-02 14:09:34.112777 7f770a7e4700 20 RGWEnv::set(): QUERY_STRING: type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:34.112778 7f770a7e4700 20 RGWEnv::set(): REMOTE_USER:
2016-06-02 14:09:34.112778 7f770a7e4700 20 RGWEnv::set(): SCRIPT_URI: /admin/log/
2016-06-02 14:09:34.112779 7f770a7e4700 20 RGWEnv::set(): SERVER_PORT: 8001
2016-06-02 14:09:34.112780 7f770a7e4700 20 HTTP_ACCEPT=*/*
2016-06-02 14:09:34.112780 7f770a7e4700 20 HTTP_AUTHORIZATION=AWS 1555b35654ad1656d804:F45z4l4bjPDDsSGmNZku0jBvHug=
2016-06-02 14:09:34.112781 7f770a7e4700 20 HTTP_DATE=Thu Jun  2 12:09:34 2016
2016-06-02 14:09:34.112781 7f770a7e4700 20 HTTP_EXPECT=100-continue
2016-06-02 14:09:34.112782 7f770a7e4700 20 HTTP_HOST=localhost:8001
2016-06-02 14:09:34.112782 7f770a7e4700 20 HTTP_TRANSFER_ENCODING=chunked
2016-06-02 14:09:34.112782 7f770a7e4700 20 QUERY_STRING=type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:34.112783 7f770a7e4700 20 REMOTE_USER=
2016-06-02 14:09:34.112783 7f770a7e4700 20 REQUEST_METHOD=GET
2016-06-02 14:09:34.112783 7f770a7e4700 20 REQUEST_URI=/admin/log/
2016-06-02 14:09:34.112784 7f770a7e4700 20 SCRIPT_URI=/admin/log/
2016-06-02 14:09:34.112784 7f770a7e4700 20 SERVER_PORT=8001
2016-06-02 14:09:34.112785 7f770a7e4700  1 ====== starting new request req=0x7f770a7de790 =====
2016-06-02 14:09:34.112799 7f770a7e4700  2 req 638:0.000013::GET /admin/log/::initializing for trans_id = tx00000000000000000027e-00575021fe-5e2e-us-east-1
2016-06-02 14:09:34.112802 7f770a7e4700 10 host=localhost
2016-06-02 14:09:34.112804 7f770a7e4700 20 subdomain= domain=localhost in_hosted_domain=1 in_hosted_domain_s3website=0
2016-06-02 14:09:34.112822 7f770a7e4700 10 handler=14RGWHandler_Log
2016-06-02 14:09:34.112824 7f770a7e4700  2 req 638:0.000039::GET /admin/log/::getting op 0
2016-06-02 14:09:34.112827 7f770a7e4700 10 op=23RGWOp_DATALog_ShardInfo
2016-06-02 14:09:34.112828 7f770a7e4700  2 req 638:0.000043::GET /admin/log/:get_data_changes_log_shard_info:authorizing
2016-06-02 14:09:34.112845 7f770a7e4700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:34.112846 7f770a7e4700 10 auth_hdr:
GET


Thu Jun  2 12:09:34 2016
/admin/log/
2016-06-02 14:09:34.112881 7f770a7e4700 15 calculated digest=F45z4l4bjPDDsSGmNZku0jBvHug=
2016-06-02 14:09:34.112883 7f770a7e4700 15 auth_sign=F45z4l4bjPDDsSGmNZku0jBvHug=
2016-06-02 14:09:34.112883 7f770a7e4700 15 compare=0
2016-06-02 14:09:34.112884 7f770a7e4700 20 system request
2016-06-02 14:09:34.112886 7f770a7e4700  2 req 638:0.000101::GET /admin/log/:get_data_changes_log_shard_info:normalizing buckets and tenants
2016-06-02 14:09:34.112898 7f770a7e4700  2 req 638:0.000102::GET /admin/log/:get_data_changes_log_shard_info:init permissions
2016-06-02 14:09:34.112900 7f770a7e4700  2 req 638:0.000115::GET /admin/log/:get_data_changes_log_shard_info:recalculating target
2016-06-02 14:09:34.112901 7f770a7e4700  2 req 638:0.000116::GET /admin/log/:get_data_changes_log_shard_info:reading permissions
2016-06-02 14:09:34.112902 7f770a7e4700  2 req 638:0.000117::GET /admin/log/:get_data_changes_log_shard_info:init op
2016-06-02 14:09:34.112903 7f770a7e4700  2 req 638:0.000117::GET /admin/log/:get_data_changes_log_shard_info:verifying op mask
2016-06-02 14:09:34.112904 7f770a7e4700 20 required_mask= 0 user.op_mask=7
2016-06-02 14:09:34.112905 7f770a7e4700  2 req 638:0.000119::GET /admin/log/:get_data_changes_log_shard_info:verifying op permissions
2016-06-02 14:09:34.112906 7f770a7e4700  2 overriding permissions due to system operation
2016-06-02 14:09:34.112906 7f770a7e4700  2 req 638:0.000121::GET /admin/log/:get_data_changes_log_shard_info:verifying op params
2016-06-02 14:09:34.112907 7f770a7e4700  2 req 638:0.000122::GET /admin/log/:get_data_changes_log_shard_info:pre-executing
2016-06-02 14:09:34.112908 7f770a7e4700  2 req 638:0.000123::GET /admin/log/:get_data_changes_log_shard_info:executing
2016-06-02 14:09:34.112944 7f770a7e4700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2627 5.f9325cdd data_log.36 [call log.info] snapc 0=[] ack+read+known_if_redirected e32) v7 -- ?+0 0x7f779c039050 con 0x5640603e9110
2016-06-02 14:09:34.113469 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1192 ==== osd_op_reply(2627 data_log.36 [call] v0'0 uv1686 ondisk = 0) v7 ==== 131+0+50 (2178453349 0 3348773465) 0x7f780c021410 con 0x5640603e9110
2016-06-02 14:09:34.113520 7f770a7e4700  2 req 638:0.000734::GET /admin/log/:get_data_changes_log_shard_info:completing
2016-06-02 14:09:34.113598 7f770a7e4700  2 req 638:0.000813::GET /admin/log/:get_data_changes_log_shard_info:op status=0
2016-06-02 14:09:34.113601 7f770a7e4700  2 req 638:0.000815::GET /admin/log/:get_data_changes_log_shard_info:http status=200
2016-06-02 14:09:34.113602 7f770a7e4700  1 ====== req done req=0x7f770a7de790 op status=0 http_status=200 ======
2016-06-02 14:09:34.113746 7f770a7e4700  1 civetweb: 0x7f779c003e30: 127.0.0.1 - - [02/Jun/2016:14:09:34 +0200] "GET /admin/log/ HTTP/1.1" 200 0 - -
2016-06-02 14:09:34.163351 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:34.163475 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:34.163568 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:34 2016
2016-06-02 14:09:34.163570 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:34 2016
2016-06-02 14:09:34.163584 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:34.163584 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:34 2016
/admin/log/
2016-06-02 14:09:34.163605 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:F45z4l4bjPDDsSGmNZku0jBvHug=
2016-06-02 14:09:34.163623 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:34.163637 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=349, easy_handle=0x7f773c380450
2016-06-02 14:09:34.163643 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:34.163679 7f77477fe700 20 link_request req_data=0x7f773c0ab8b0 req_data->id=349, easy_handle=0x7f773c380450
2016-06-02 14:09:34.164513 7f77477fe700 10 receive_http_header
2016-06-02 14:09:34.164517 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:34.164519 7f77477fe700 10 receive_http_header
2016-06-02 14:09:34.164519 7f77477fe700 10 received header:x-amz-request-id: tx000000000000000000202-00575021fe-3728-us-west
2016-06-02 14:09:34.164532 7f77477fe700 10 receive_http_header
2016-06-02 14:09:34.164533 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:34.164535 7f77477fe700 10 receive_http_header
2016-06-02 14:09:34.164535 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:34 GMT
2016-06-02 14:09:34.164537 7f77477fe700 10 receive_http_header
2016-06-02 14:09:34.164537 7f77477fe700 10 received header:
2016-06-02 14:09:34.164611 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:34.164970 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:34.164974 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:34.164976 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:34.164991 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:34.397256 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2628 2.4322fa9f notify.0 [watch ping cookie 94834492702224] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f800ac30 con 0x5640603ecb10
2016-06-02 14:09:34.397307 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2629 2.88aa5c95 notify.2 [watch ping cookie 94834492706960] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f8019ba0 con 0x5640603ecb10
2016-06-02 14:09:34.397323 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2630 2.f8c99aee notify.3 [watch ping cookie 94834492709872] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f801acc0 con 0x5640603ecb10
2016-06-02 14:09:34.397329 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2631 2.a204812d notify.4 [watch ping cookie 94834492712752] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f8018880 con 0x5640603ecb10
2016-06-02 14:09:34.397419 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2632 2.31099063 notify.5 [watch ping cookie 94834492715648] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f801b490 con 0x5640603ecb10
2016-06-02 14:09:34.397445 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2633 2.97c520d4 notify.6 [watch ping cookie 94834492718368] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f801c490 con 0x5640603ecb10
2016-06-02 14:09:34.397456 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2634 2.16dafda0 notify.1 [watch ping cookie 94834492703856] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f801acc0 con 0x5640603f1eb0
2016-06-02 14:09:34.397469 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2635 2.84ada7c9 notify.7 [watch ping cookie 94834492721344] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77f801c7b0 con 0x5640603f1eb0
2016-06-02 14:09:34.397482 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- ping magic: 0 v1 -- ?+0 0x7f77f801b490 con 0x5640603ecb10
2016-06-02 14:09:34.397492 7f783940e700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- ping magic: 0 v1 -- ?+0 0x7f77f801c490 con 0x5640603f1eb0
2016-06-02 14:09:34.397792 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 902 ==== osd_op_reply(2634 notify.1 [watch ping cookie 94834492703856] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (689729908 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:34.397793 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 681 ==== osd_op_reply(2629 notify.2 [watch ping cookie 94834492706960] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3592388915 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:34.397865 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 682 ==== osd_op_reply(2630 notify.3 [watch ping cookie 94834492709872] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1695028518 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:34.397894 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 903 ==== osd_op_reply(2635 notify.7 [watch ping cookie 94834492721344] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1842238908 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:34.397901 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 683 ==== osd_op_reply(2631 notify.4 [watch ping cookie 94834492712752] v0'0 uv3 ondisk = 0) v7 ==== 128+0+0 (1823463381 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:34.397919 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 684 ==== osd_op_reply(2628 notify.0 [watch ping cookie 94834492702224] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (4158488181 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:34.398073 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 685 ==== osd_op_reply(2632 notify.5 [watch ping cookie 94834492715648] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3716607297 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:34.398115 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 686 ==== osd_op_reply(2633 notify.6 [watch ping cookie 94834492718368] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1447217987 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:34.502229 7f76e6f9d700  1 -- 127.0.0.1:0/2056988180 --> 127.0.0.1:6808/30507 -- osd_op(client.24113.0:12 1.fd9d5d69 realms.00b17c78-da61-464b-906a-81dbe28b899d.control [watch ping cookie 94834493713904] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f77fc026620 con 0x5640604f3290
2016-06-02 14:09:34.502274 7f76e6f9d700  1 -- 127.0.0.1:0/2056988180 --> 127.0.0.1:6808/30507 -- ping magic: 0 v1 -- ?+0 0x7f77fc0271d0 con 0x5640604f3290
2016-06-02 14:09:34.502909 7f78281f2700  1 -- 127.0.0.1:0/2056988180 <== osd.2 127.0.0.1:6808/30507 12 ==== osd_op_reply(12 realms.00b17c78-da61-464b-906a-81dbe28b899d.control [watch ping cookie 94834493713904] v0'0 uv1 ondisk = 0) v7 ==== 171+0+0 (3084123279 0 0) 0x7f77e801def0 con 0x5640604f3290
2016-06-02 14:09:35.165191 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:35.165307 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:35.165388 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:35 2016
2016-06-02 14:09:35.165392 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:35 2016
2016-06-02 14:09:35.165398 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:35.165399 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:35 2016
/admin/log/
2016-06-02 14:09:35.165431 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:Mg16+5sNL12jovodOD+PYe9jG4M=
2016-06-02 14:09:35.165443 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:35.165452 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=350, easy_handle=0x7f773c380450
2016-06-02 14:09:35.165459 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:35.165483 7f77477fe700 20 link_request req_data=0x7f773c0e84e0 req_data->id=350, easy_handle=0x7f773c380450
2016-06-02 14:09:35.166386 7f77477fe700 10 receive_http_header
2016-06-02 14:09:35.166390 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:35.166392 7f77477fe700 10 receive_http_header
2016-06-02 14:09:35.166393 7f77477fe700 10 received header:x-amz-request-id: tx000000000000000000203-00575021ff-3728-us-west
2016-06-02 14:09:35.166406 7f77477fe700 10 receive_http_header
2016-06-02 14:09:35.166407 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:35.166409 7f77477fe700 10 receive_http_header
2016-06-02 14:09:35.166409 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:35 GMT
2016-06-02 14:09:35.166411 7f77477fe700 10 receive_http_header
2016-06-02 14:09:35.166412 7f77477fe700 10 received header:
2016-06-02 14:09:35.166441 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:35.166825 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:35.166832 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:35.166844 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:35.166850 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:36.385882 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:36.386028 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:36.386134 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:36 2016
2016-06-02 14:09:36.386137 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:36 2016
2016-06-02 14:09:36.386151 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:36.386152 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:36 2016
/admin/log/
2016-06-02 14:09:36.386195 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:aDsAVBPeWCNh2FJ1UOvLwuzLFck=
2016-06-02 14:09:36.386216 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:36.386222 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=351, easy_handle=0x7f773c380450
2016-06-02 14:09:36.386229 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:36.386255 7f77477fe700 20 link_request req_data=0x7f773c315010 req_data->id=351, easy_handle=0x7f773c380450
2016-06-02 14:09:36.387263 7f77477fe700 10 receive_http_header
2016-06-02 14:09:36.387268 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:36.387270 7f77477fe700 10 receive_http_header
2016-06-02 14:09:36.387271 7f77477fe700 10 received header:x-amz-request-id: tx000000000000000000204-0057502200-3728-us-west
2016-06-02 14:09:36.387275 7f77477fe700 10 receive_http_header
2016-06-02 14:09:36.387276 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:36.387277 7f77477fe700 10 receive_http_header
2016-06-02 14:09:36.387277 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:36 GMT
2016-06-02 14:09:36.387279 7f77477fe700 10 receive_http_header
2016-06-02 14:09:36.387279 7f77477fe700 10 received header:
2016-06-02 14:09:36.387309 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:36.387662 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:36.387668 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:36.387672 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:36.387677 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:37.169332 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:37.169465 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:37.169562 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:37 2016
2016-06-02 14:09:37.169566 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:37 2016
2016-06-02 14:09:37.169580 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:37.169581 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:37 2016
/admin/log/
2016-06-02 14:09:37.169610 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:j8LJtZNEZjo+jkgWyVrDGhACoj0=
2016-06-02 14:09:37.169629 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:37.169646 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=352, easy_handle=0x7f773c380450
2016-06-02 14:09:37.169653 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:37.169678 7f77477fe700 20 link_request req_data=0x7f773c08e4e0 req_data->id=352, easy_handle=0x7f773c380450
2016-06-02 14:09:37.170734 7f77477fe700 10 receive_http_header
2016-06-02 14:09:37.170739 7f77477fe700 10 received header:HTTP/1.1 200 OK
2016-06-02 14:09:37.170741 7f77477fe700 10 receive_http_header
2016-06-02 14:09:37.170743 7f77477fe700 10 received header:x-amz-request-id: tx000000000000000000205-0057502201-3728-us-west
2016-06-02 14:09:37.170747 7f77477fe700 10 receive_http_header
2016-06-02 14:09:37.170748 7f77477fe700 10 received header:Content-Length: 83
2016-06-02 14:09:37.170750 7f77477fe700 10 receive_http_header
2016-06-02 14:09:37.170751 7f77477fe700 10 received header:Date: Thu, 02 Jun 2016 12:09:37 GMT
2016-06-02 14:09:37.170753 7f77477fe700 10 receive_http_header
2016-06-02 14:09:37.170753 7f77477fe700 10 received header:
2016-06-02 14:09:37.170789 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:37.171129 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:37.171134 7f7760ff9700 20 incremental_sync:1057: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:37.171137 7f7760ff9700 20 incremental_sync:1104: shard_id=36 datalog_marker=1_1464869364.852692_3801.1 sync_marker.marker=1_1464869364.852692_3801.1
2016-06-02 14:09:37.171144 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:38.031605 7f7744ff9700  1 handle_sigterm
2016-06-02 14:09:38.031619 7f7744ff9700  1 handle_sigterm set alarm for 120
2016-06-02 14:09:38.031620 7f7855fa2ac0 -1 shutting down
2016-06-02 14:09:38.172131 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:38.172245 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:38.172328 7f7760ff9700 20 RGWEnv::set(): HTTP_DATE: Thu Jun  2 12:09:38 2016
2016-06-02 14:09:38.172331 7f7760ff9700 20 > HTTP_DATE -> Thu Jun  2 12:09:38 2016
2016-06-02 14:09:38.172336 7f7760ff9700 10 get_canon_resource(): dest=/admin/log/
2016-06-02 14:09:38.172336 7f7760ff9700 10 generated canonical header: GET


Thu Jun  2 12:09:38 2016
/admin/log/
2016-06-02 14:09:38.172371 7f7760ff9700 15 generated auth header: AWS 1555b35654ad1656d804:Q2DhPxM5NxYiteNIEShcA8O1G8Q=
2016-06-02 14:09:38.172381 7f7760ff9700 20 sending request to http://localhost:8002/admin/log/?type=data&id=36&info&rgwx-zonegroup=9e28164a-9dc5-4a79-8932-2fe63613d26a
2016-06-02 14:09:38.172388 7f7760ff9700 20 register_request mgr=0x5640604329f8 req_data->id=353, easy_handle=0x7f773c380450
2016-06-02 14:09:38.172395 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:38.172418 7f77477fe700 20 link_request req_data=0x7f773c2c5db0 req_data->id=353, easy_handle=0x7f773c380450
2016-06-02 14:09:38.172978 7f77477fe700 20 ERROR: msg->data.result=7 req_data->id=353 http_status=0
2016-06-02 14:09:38.172983 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate()
2016-06-02 14:09:38.172988 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c389ee0:31RGWReadRemoteDataLogShardInfoCR: operate() returned r=-11
2016-06-02 14:09:38.173314 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:38.173319 7f7760ff9700  0 ERROR: failed to fetch remote data log info: ret=-11
2016-06-02 14:09:38.173325 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:38.173327 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c26d9b0 is still running
2016-06-02 14:09:38.173328 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c0a44c0 is complete
2016-06-02 14:09:38.173330 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c0ac590 is complete
2016-06-02 14:09:38.173331 7f7760ff9700 20 run: stack=0x7f773c182de0 is_blocked_by_stack()=0 is_sleeping=0 waiting_for_child()=1
2016-06-02 14:09:38.173334 7f7760ff9700 20 cr:s=0x7f773c26d9b0:op=0x7f773c26cf90:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:38.173374 7f7760ff9700 20 cr:s=0x7f773c26d9b0:op=0x7f773c136d60:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:38.173377 7f7760ff9700 20 cr:s=0x7f773c26d9b0:op=0x7f773c136d60:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:38.173395 7f7760ff9700 20 enqueued request req=0x7f773c2ec170
2016-06-02 14:09:38.173397 7f7760ff9700 20 RGWWQ:
2016-06-02 14:09:38.173397 7f7760ff9700 20 req: 0x7f773c2ec170
2016-06-02 14:09:38.173401 7f7760ff9700 20 run: stack=0x7f773c26d9b0 is io blocked
2016-06-02 14:09:38.173404 7f77b1ffb700 20 dequeued request req=0x7f773c2ec170
2016-06-02 14:09:38.173407 7f77b1ffb700 20 RGWWQ: empty
2016-06-02 14:09:38.173445 7f77b1ffb700  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6800/30487 -- osd_op(client.24110.0:2636 5.c8d946cb datalog.sync-status.shard.91441c60-0ca1-4118-9685-990e04272142.36 [call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x7f777c01b240 con 0x5640603e9110
2016-06-02 14:09:38.183113 7f783820a700  1 -- 127.0.0.1:0/2000881652 <== osd.0 127.0.0.1:6800/30487 1193 ==== osd_op_reply(2636 datalog.sync-status.shard.91441c60-0ca1-4118-9685-990e04272142.36 [call] v32'1888 uv1888 ondisk = 0) v7 ==== 185+0+0 (2930197259 0 0) 0x7f780c029a40 con 0x5640603e9110
2016-06-02 14:09:38.183208 7f7760ff9700 20 cr:s=0x7f773c26d9b0:op=0x7f773c136d60:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:38.183216 7f7760ff9700 20 cr:s=0x7f773c26d9b0:op=0x7f773c136d60:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:38.183218 7f7760ff9700 20 cr:s=0x7f773c26d9b0:op=0x7f773c136d60:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:38.183219 7f7760ff9700 20 cr:s=0x7f773c26d9b0:op=0x7f773c136d60:22RGWSimpleRadosUnlockCR: operate()
2016-06-02 14:09:38.183513 7f7760ff9700 20 cr:s=0x7f773c26d9b0:op=0x7f773c26cf90:20RGWContinuousLeaseCR: operate()
2016-06-02 14:09:38.183517 7f7760ff9700 20 run: stack=0x7f773c26d9b0 is done
2016-06-02 14:09:38.183521 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:38.183523 7f7760ff9700 20 collect(): s=0x7f773c182de0 stack=0x7f773c26d9b0 is complete
2016-06-02 14:09:38.183525 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate()
2016-06-02 14:09:38.183526 7f7760ff9700 10 sync: incremental_sync: shard_id=36 r=-11
2016-06-02 14:09:38.183528 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c1fd4c0:18RGWDataSyncShardCR: operate() returned r=-11
2016-06-02 14:09:38.183530 7f7760ff9700 20 cr:s=0x7f773c182de0:op=0x7f773c04fe70:25RGWDataSyncShardControlCR: operate()
2016-06-02 14:09:38.183548 7f7760ff9700 20 run: stack=0x7f773c182de0 is io blocked
2016-06-02 14:09:38.213656 7f7760ff9700  0 ERROR: failed to clone shard, completion_mgr.get_next() returned ret=-125
2016-06-02 14:09:38.213676 7f7760ff9700  5 run(): was stopped, exiting
2016-06-02 14:09:38.213678 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c00f110 nref=2
2016-06-02 14:09:38.213680 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c013250 nref=2
2016-06-02 14:09:38.213681 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0142d0 nref=2
2016-06-02 14:09:38.213681 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c01d810 nref=2
2016-06-02 14:09:38.213682 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c03ae10 nref=2
2016-06-02 14:09:38.213682 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c049bc0 nref=2
2016-06-02 14:09:38.213683 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c04a970 nref=2
2016-06-02 14:09:38.213684 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c04efc0 nref=2
2016-06-02 14:09:38.213685 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c050aa0 nref=2
2016-06-02 14:09:38.213685 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c052430 nref=2
2016-06-02 14:09:38.213686 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c056a70 nref=2
2016-06-02 14:09:38.213687 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c05d810 nref=2
2016-06-02 14:09:38.213703 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c063820 nref=2
2016-06-02 14:09:38.213704 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c06dee0 nref=2
2016-06-02 14:09:38.213705 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c06fa20 nref=2
2016-06-02 14:09:38.213706 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0707f0 nref=2
2016-06-02 14:09:38.213706 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0715a0 nref=2
2016-06-02 14:09:38.213707 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c074c70 nref=2
2016-06-02 14:09:38.213707 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c075a40 nref=2
2016-06-02 14:09:38.213709 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c07c7f0 nref=2
2016-06-02 14:09:38.213709 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0887f0 nref=2
2016-06-02 14:09:38.213710 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c089580 nref=2
2016-06-02 14:09:38.213711 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c08e7f0 nref=2
2016-06-02 14:09:38.213712 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c093a40 nref=2
2016-06-02 14:09:38.213713 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0947f0 nref=2
2016-06-02 14:09:38.213714 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c097110 nref=2
2016-06-02 14:09:38.213719 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0b0540 nref=2
2016-06-02 14:09:38.213720 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0b12b0 nref=2
2016-06-02 14:09:38.213721 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0b4520 nref=2
2016-06-02 14:09:38.213721 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0b8770 nref=2
2016-06-02 14:09:38.213722 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0b9650 nref=2
2016-06-02 14:09:38.213723 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0bc660 nref=2
2016-06-02 14:09:38.213724 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0bf6c0 nref=2
2016-06-02 14:09:38.213725 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0c23b0 nref=2
2016-06-02 14:09:38.213725 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0c8410 nref=2
2016-06-02 14:09:38.213726 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0cb6f0 nref=2
2016-06-02 14:09:38.213727 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0cd770 nref=2
2016-06-02 14:09:38.213728 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0cffa0 nref=2
2016-06-02 14:09:38.213729 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0dc310 nref=2
2016-06-02 14:09:38.213730 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0dce70 nref=2
2016-06-02 14:09:38.213730 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0ddac0 nref=2
2016-06-02 14:09:38.213731 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0dea10 nref=1
2016-06-02 14:09:38.213750 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0e0710 nref=2
2016-06-02 14:09:38.213750 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0e4b70 nref=2
2016-06-02 14:09:38.213751 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0ecc50 nref=2
2016-06-02 14:09:38.213752 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0ee4d0 nref=2
2016-06-02 14:09:38.213752 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0efe20 nref=2
2016-06-02 14:09:38.213753 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c0fe8e0 nref=2
2016-06-02 14:09:38.213754 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c106d40 nref=2
2016-06-02 14:09:38.213754 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c109210 nref=2
2016-06-02 14:09:38.213755 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c10bda0 nref=2
2016-06-02 14:09:38.213755 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c121240 nref=2
2016-06-02 14:09:38.213756 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c12ad60 nref=2
2016-06-02 14:09:38.213757 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c12f7b0 nref=2
2016-06-02 14:09:38.213758 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c131e50 nref=2
2016-06-02 14:09:38.213758 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c13b0f0 nref=2
2016-06-02 14:09:38.213759 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c13d0a0 nref=2
2016-06-02 14:09:38.213759 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c141670 nref=2
2016-06-02 14:09:38.213760 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c141c60 nref=2
2016-06-02 14:09:38.213761 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c174c30 nref=2
2016-06-02 14:09:38.213761 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c176e00 nref=2
2016-06-02 14:09:38.213762 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c178490 nref=2
2016-06-02 14:09:38.213762 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c179540 nref=2
2016-06-02 14:09:38.213763 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c17afc0 nref=2
2016-06-02 14:09:38.213764 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c17baf0 nref=2
2016-06-02 14:09:38.213764 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c17d1b0 nref=2
2016-06-02 14:09:38.213765 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c17dcd0 nref=2
2016-06-02 14:09:38.213766 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c17f390 nref=2
2016-06-02 14:09:38.213766 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c17fed0 nref=2
2016-06-02 14:09:38.213767 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c180a20 nref=2
2016-06-02 14:09:38.213767 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c182060 nref=2
2016-06-02 14:09:38.213768 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c182de0 nref=2
2016-06-02 14:09:38.213769 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1842a0 nref=2
2016-06-02 14:09:38.213769 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1858c0 nref=2
2016-06-02 14:09:38.213770 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c186470 nref=2
2016-06-02 14:09:38.213771 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c186fb0 nref=2
2016-06-02 14:09:38.213771 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c187b00 nref=2
2016-06-02 14:09:38.213772 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c189190 nref=2
2016-06-02 14:09:38.213772 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c189cf0 nref=2
2016-06-02 14:09:38.213773 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c18b380 nref=2
2016-06-02 14:09:38.213774 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c18e140 nref=2
2016-06-02 14:09:38.213774 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c18ec70 nref=2
2016-06-02 14:09:38.213775 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c190340 nref=2
2016-06-02 14:09:38.213776 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1919e0 nref=2
2016-06-02 14:09:38.213777 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c193000 nref=2
2016-06-02 14:09:38.213778 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c194c70 nref=2
2016-06-02 14:09:38.213779 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c196070 nref=2
2016-06-02 14:09:38.213779 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1981d0 nref=2
2016-06-02 14:09:38.213780 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c19a3a0 nref=2
2016-06-02 14:09:38.213780 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c19aee0 nref=2
2016-06-02 14:09:38.213781 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c19d1a0 nref=2
2016-06-02 14:09:38.213781 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c19d330 nref=2
2016-06-02 14:09:38.213782 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c19dc80 nref=2
2016-06-02 14:09:38.213782 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c19fe90 nref=2
2016-06-02 14:09:38.213783 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1a0a00 nref=2
2016-06-02 14:09:38.213784 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1a2060 nref=2
2016-06-02 14:09:38.213784 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1a2b70 nref=2
2016-06-02 14:09:38.213785 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1a4320 nref=2
2016-06-02 14:09:38.213786 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1a5960 nref=2
2016-06-02 14:09:38.213786 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1a6ff0 nref=2
2016-06-02 14:09:38.213787 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1a71b0 nref=2
2016-06-02 14:09:38.213788 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1a7b10 nref=2
2016-06-02 14:09:38.213789 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1aa9f0 nref=2
2016-06-02 14:09:38.213789 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1ad330 nref=2
2016-06-02 14:09:38.213790 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1aedb0 nref=2
2016-06-02 14:09:38.213790 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1af520 nref=2
2016-06-02 14:09:38.213791 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1b0440 nref=2
2016-06-02 14:09:38.213792 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1b0fb0 nref=2
2016-06-02 14:09:38.213792 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1b1b00 nref=2
2016-06-02 14:09:38.213793 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1b3190 nref=2
2016-06-02 14:09:38.213794 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1b3cc0 nref=2
2016-06-02 14:09:38.213794 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1b69b0 nref=2
2016-06-02 14:09:38.213795 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1b8080 nref=2
2016-06-02 14:09:38.213795 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1b9310 nref=2
2016-06-02 14:09:38.213796 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1ba270 nref=2
2016-06-02 14:09:38.213797 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1badc0 nref=2
2016-06-02 14:09:38.213798 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1baf80 nref=2
2016-06-02 14:09:38.213798 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1bb500 nref=2
2016-06-02 14:09:38.213799 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1bb9d0 nref=2
2016-06-02 14:09:38.213799 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1bbaf0 nref=2
2016-06-02 14:09:38.213800 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1bcf80 nref=2
2016-06-02 14:09:38.213801 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1bd140 nref=2
2016-06-02 14:09:38.213802 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1bda80 nref=2
2016-06-02 14:09:38.213802 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1be230 nref=2
2016-06-02 14:09:38.213803 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1bf160 nref=2
2016-06-02 14:09:38.213804 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1bf350 nref=2
2016-06-02 14:09:38.213804 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1bfca0 nref=2
2016-06-02 14:09:38.213805 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1c1320 nref=2
2016-06-02 14:09:38.213805 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c1c2bf0 nref=2
2016-06-02 14:09:38.213806 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c24d270 nref=2
2016-06-02 14:09:38.213807 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c24e110 nref=2
2016-06-02 14:09:38.213807 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c24efb0 nref=2
2016-06-02 14:09:38.213808 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c24fe30 nref=2
2016-06-02 14:09:38.213808 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c250ca0 nref=2
2016-06-02 14:09:38.213810 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c251b10 nref=2
2016-06-02 14:09:38.213810 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c252980 nref=2
2016-06-02 14:09:38.213811 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2537f0 nref=2
2016-06-02 14:09:38.213812 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c254660 nref=2
2016-06-02 14:09:38.213813 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2554d0 nref=2
2016-06-02 14:09:38.213813 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c256340 nref=2
2016-06-02 14:09:38.213814 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c257170 nref=2
2016-06-02 14:09:38.213815 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c257ff0 nref=2
2016-06-02 14:09:38.213816 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c258eb0 nref=2
2016-06-02 14:09:38.213817 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c259d20 nref=2
2016-06-02 14:09:38.213817 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c25ab90 nref=2
2016-06-02 14:09:38.213818 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c25b9f0 nref=2
2016-06-02 14:09:38.213818 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c25c850 nref=2
2016-06-02 14:09:38.213819 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c25d6b0 nref=2
2016-06-02 14:09:38.213820 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c25e510 nref=2
2016-06-02 14:09:38.213820 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c25f370 nref=2
2016-06-02 14:09:38.213821 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c260190 nref=2
2016-06-02 14:09:38.213821 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c261000 nref=2
2016-06-02 14:09:38.213822 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c261eb0 nref=2
2016-06-02 14:09:38.213823 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c262d10 nref=2
2016-06-02 14:09:38.213823 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c263b70 nref=2
2016-06-02 14:09:38.213824 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2649d0 nref=2
2016-06-02 14:09:38.213825 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c265830 nref=2
2016-06-02 14:09:38.213825 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c266690 nref=2
2016-06-02 14:09:38.213826 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2674f0 nref=2
2016-06-02 14:09:38.213826 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c268350 nref=2
2016-06-02 14:09:38.213827 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c269170 nref=2
2016-06-02 14:09:38.213828 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c269fe0 nref=2
2016-06-02 14:09:38.213829 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c26ae90 nref=2
2016-06-02 14:09:38.213829 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c26bcf0 nref=2
2016-06-02 14:09:38.213830 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c26cb50 nref=2
2016-06-02 14:09:38.213830 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c26e810 nref=2
2016-06-02 14:09:38.213831 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c26f670 nref=2
2016-06-02 14:09:38.213832 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2704d0 nref=2
2016-06-02 14:09:38.213833 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c271330 nref=2
2016-06-02 14:09:38.213834 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c272150 nref=2
2016-06-02 14:09:38.213835 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c272fe0 nref=2
2016-06-02 14:09:38.213835 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c273e70 nref=2
2016-06-02 14:09:38.213836 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c274cd0 nref=2
2016-06-02 14:09:38.213837 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c275b30 nref=2
2016-06-02 14:09:38.213837 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c276990 nref=2
2016-06-02 14:09:38.213838 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2777f0 nref=2
2016-06-02 14:09:38.213838 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c278650 nref=2
2016-06-02 14:09:38.213839 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2794b0 nref=2
2016-06-02 14:09:38.213840 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c27a310 nref=2
2016-06-02 14:09:38.213840 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c27b130 nref=2
2016-06-02 14:09:38.213841 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c27bfc0 nref=2
2016-06-02 14:09:38.213842 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c27ce50 nref=2
2016-06-02 14:09:38.213842 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c27dcb0 nref=2
2016-06-02 14:09:38.213843 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c27eb00 nref=2
2016-06-02 14:09:38.213843 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c27f970 nref=2
2016-06-02 14:09:38.213844 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2807e0 nref=2
2016-06-02 14:09:38.213845 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c281650 nref=2
2016-06-02 14:09:38.213845 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2824c0 nref=2
2016-06-02 14:09:38.213846 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c283330 nref=2
2016-06-02 14:09:38.213847 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c284160 nref=2
2016-06-02 14:09:38.213848 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c284fe0 nref=2
2016-06-02 14:09:38.213848 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c285e90 nref=2
2016-06-02 14:09:38.213849 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c286cf0 nref=2
2016-06-02 14:09:38.213850 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c287b50 nref=2
2016-06-02 14:09:38.213850 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2889b0 nref=2
2016-06-02 14:09:38.213850 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c289810 nref=2
2016-06-02 14:09:38.213851 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c28a670 nref=2
2016-06-02 14:09:38.213852 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c28b4d0 nref=2
2016-06-02 14:09:38.213852 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c28c330 nref=2
2016-06-02 14:09:38.213853 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c28d150 nref=2
2016-06-02 14:09:38.213854 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c28dfe0 nref=2
2016-06-02 14:09:38.213855 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c28ee70 nref=2
2016-06-02 14:09:38.213855 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c28fcd0 nref=2
2016-06-02 14:09:38.213856 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c290b30 nref=2
2016-06-02 14:09:38.213856 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c291990 nref=2
2016-06-02 14:09:38.213857 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2927f0 nref=2
2016-06-02 14:09:38.213858 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c293640 nref=2
2016-06-02 14:09:38.213859 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2944b0 nref=2
2016-06-02 14:09:38.213859 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c295320 nref=2
2016-06-02 14:09:38.213860 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c296150 nref=2
2016-06-02 14:09:38.213860 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c296ff0 nref=2
2016-06-02 14:09:38.213861 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c297e80 nref=2
2016-06-02 14:09:38.213862 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c298ce0 nref=2
2016-06-02 14:09:38.213862 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c299b40 nref=2
2016-06-02 14:09:38.213863 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c29a9a0 nref=2
2016-06-02 14:09:38.213863 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c29b800 nref=2
2016-06-02 14:09:38.213864 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c29c660 nref=2
2016-06-02 14:09:38.213865 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c29d4c0 nref=2
2016-06-02 14:09:38.213866 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c29e320 nref=2
2016-06-02 14:09:38.213866 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c29f140 nref=2
2016-06-02 14:09:38.213867 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c29ffd0 nref=2
2016-06-02 14:09:38.213868 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2a0e60 nref=2
2016-06-02 14:09:38.213868 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2a1cc0 nref=2
2016-06-02 14:09:38.213869 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2a2b20 nref=2
2016-06-02 14:09:38.213869 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2a3980 nref=2
2016-06-02 14:09:38.213870 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2a47e0 nref=2
2016-06-02 14:09:38.213871 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2a5640 nref=2
2016-06-02 14:09:38.213871 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2a64a0 nref=2
2016-06-02 14:09:38.213872 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2a7300 nref=2
2016-06-02 14:09:38.213873 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2a8120 nref=2
2016-06-02 14:09:38.213873 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2a8fb0 nref=2
2016-06-02 14:09:38.213874 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2a9e40 nref=2
2016-06-02 14:09:38.213874 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2aaca0 nref=2
2016-06-02 14:09:38.213875 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2abb00 nref=2
2016-06-02 14:09:38.213876 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2ac960 nref=2
2016-06-02 14:09:38.213877 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2ad7c0 nref=2
2016-06-02 14:09:38.213878 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2ae620 nref=2
2016-06-02 14:09:38.213878 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2af480 nref=2
2016-06-02 14:09:38.213879 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2b02e0 nref=2
2016-06-02 14:09:38.213879 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2b1100 nref=2
2016-06-02 14:09:38.213880 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2b1f90 nref=2
2016-06-02 14:09:38.213881 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2b2e20 nref=2
2016-06-02 14:09:38.213882 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2b3c80 nref=2
2016-06-02 14:09:38.213882 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2b4ae0 nref=2
2016-06-02 14:09:38.213883 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2b5940 nref=2
2016-06-02 14:09:38.213883 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2b67a0 nref=2
2016-06-02 14:09:38.213884 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2b75f0 nref=2
2016-06-02 14:09:38.213885 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2b8460 nref=2
2016-06-02 14:09:38.213885 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2b92d0 nref=2
2016-06-02 14:09:38.213886 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2ba100 nref=2
2016-06-02 14:09:38.213887 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2bafa0 nref=2
2016-06-02 14:09:38.213887 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2bbe30 nref=2
2016-06-02 14:09:38.213888 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2bcc90 nref=2
2016-06-02 14:09:38.213889 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2bdaf0 nref=2
2016-06-02 14:09:38.213889 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2be950 nref=2
2016-06-02 14:09:38.213890 7f7760ff9700 20 clearing stack on run() exit: stack=0x7f773c2bf7b0 nref=2
2016-06-02 14:09:38.213908 7f7760ff9700 20 run(stacks) returned r=-125
2016-06-02 14:09:38.228110 7f7760ff9700  0 ERROR: failed to run sync
2016-06-02 14:09:38.229493 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2637 2.4322fa9f notify.0 [watch unwatch cookie 94834492702224] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x5640604f3610 con 0x5640603ecb10
2016-06-02 14:09:38.238956 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 687 ==== osd_op_reply(2637 notify.0 [watch unwatch cookie 94834492702224] v32'77 uv1 ondisk = 0) v7 ==== 128+0+0 (1246221461 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:38.238992 7f7855fa2ac0 20 remove_watcher() i=0
2016-06-02 14:09:38.238998 7f7855fa2ac0  2 removed watcher, disabling cache
2016-06-02 14:09:38.239232 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2638 2.16dafda0 notify.1 [watch unwatch cookie 94834492703856] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x5640604f3610 con 0x5640603f1eb0
2016-06-02 14:09:38.246511 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 904 ==== osd_op_reply(2638 notify.1 [watch unwatch cookie 94834492703856] v32'77 uv1 ondisk = 0) v7 ==== 128+0+0 (2491866004 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:38.246537 7f7855fa2ac0 20 remove_watcher() i=1
2016-06-02 14:09:38.246555 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2639 2.88aa5c95 notify.2 [watch unwatch cookie 94834492706960] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x5640604f3610 con 0x5640603ecb10
2016-06-02 14:09:38.253402 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 688 ==== osd_op_reply(2639 notify.2 [watch unwatch cookie 94834492706960] v32'153 uv1 ondisk = 0) v7 ==== 128+0+0 (1526145602 0 0) 0x7f7804008080 con 0x5640603ecb10
2016-06-02 14:09:38.253426 7f7855fa2ac0 20 remove_watcher() i=2
2016-06-02 14:09:38.253441 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2640 2.f8c99aee notify.3 [watch unwatch cookie 94834492709872] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x5640604f3610 con 0x5640603ecb10
2016-06-02 14:09:38.260697 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 689 ==== osd_op_reply(2640 notify.3 [watch unwatch cookie 94834492709872] v32'77 uv1 ondisk = 0) v7 ==== 128+0+0 (3633496006 0 0) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:38.260728 7f7855fa2ac0 20 remove_watcher() i=3
2016-06-02 14:09:38.260746 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2641 2.a204812d notify.4 [watch unwatch cookie 94834492712752] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x5640604f3610 con 0x5640603ecb10
2016-06-02 14:09:38.267719 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 690 ==== osd_op_reply(2641 notify.4 [watch unwatch cookie 94834492712752] v32'154 uv3 ondisk = 0) v7 ==== 128+0+0 (4149867155 0 0) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:38.267748 7f7855fa2ac0 20 remove_watcher() i=4
2016-06-02 14:09:38.267763 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2642 2.31099063 notify.5 [watch unwatch cookie 94834492715648] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x5640604f3610 con 0x5640603ecb10
2016-06-02 14:09:38.274989 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 691 ==== osd_op_reply(2642 notify.5 [watch unwatch cookie 94834492715648] v32'77 uv1 ondisk = 0) v7 ==== 128+0+0 (1612466081 0 0) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:38.275025 7f7855fa2ac0 20 remove_watcher() i=5
2016-06-02 14:09:38.275055 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6804/30491 -- osd_op(client.24110.0:2643 2.97c520d4 notify.6 [watch unwatch cookie 94834492718368] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x5640604f3610 con 0x5640603ecb10
2016-06-02 14:09:38.282348 7f78287f8700  1 -- 127.0.0.1:0/2000881652 <== osd.1 127.0.0.1:6804/30491 692 ==== osd_op_reply(2643 notify.6 [watch unwatch cookie 94834492718368] v32'77 uv1 ondisk = 0) v7 ==== 128+0+0 (3956826531 0 0) 0x7f7804007340 con 0x5640603ecb10
2016-06-02 14:09:38.282381 7f7855fa2ac0 20 remove_watcher() i=6
2016-06-02 14:09:38.282400 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 --> 127.0.0.1:6808/30507 -- osd_op(client.24110.0:2644 2.84ada7c9 notify.7 [watch unwatch cookie 94834492721344] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x5640604f3610 con 0x5640603f1eb0
2016-06-02 14:09:38.289113 7f78285f6700  1 -- 127.0.0.1:0/2000881652 <== osd.2 127.0.0.1:6808/30507 905 ==== osd_op_reply(2644 notify.7 [watch unwatch cookie 94834492721344] v32'77 uv1 ondisk = 0) v7 ==== 128+0+0 (3495231324 0 0) 0x7f77fc0213c0 con 0x5640603f1eb0
2016-06-02 14:09:38.289162 7f7855fa2ac0 20 remove_watcher() i=7
2016-06-02 14:09:38.291489 7f7745ffb700 20 BucketsSyncThread: done
2016-06-02 14:09:38.291700 7f77457fa700 20 UserSyncThread: done
2016-06-02 14:09:38.292693 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 mark_down 0x5640603e9110 -- 0x5640603e7e90
2016-06-02 14:09:38.292778 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 mark_down 0x5640603ecb10 -- 0x5640603eb860
2016-06-02 14:09:38.292805 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 mark_down 0x5640603f1eb0 -- 0x5640603f4e00
2016-06-02 14:09:38.292845 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 mark_down 0x5640603d8b10 -- 0x5640603d8c60
2016-06-02 14:09:38.293017 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 mark_down_all
2016-06-02 14:09:38.293535 7f7855fa2ac0  1 -- 127.0.0.1:0/2000881652 shutdown complete.
2016-06-02 14:09:38.295275 7f7855fa2ac0  1 final shutdown
2016-06-02 14:09:38.295354 7f7855fa2ac0  1 -- 127.0.0.1:0/2056988180 --> 127.0.0.1:6808/30507 -- osd_op(client.24113.0:13 1.fd9d5d69 realms.00b17c78-da61-464b-906a-81dbe28b899d.control [watch unwatch cookie 94834493713904] snapc 0=[] ondisk+write+known_if_redirected e32) v7 -- ?+0 0x5640604f3610 con 0x5640604f3290
2016-06-02 14:09:38.301949 7f78281f2700  1 -- 127.0.0.1:0/2056988180 <== osd.2 127.0.0.1:6808/30507 13 ==== osd_op_reply(13 realms.00b17c78-da61-464b-906a-81dbe28b899d.control [watch unwatch cookie 94834493713904] v32'19 uv1 ondisk = 0) v7 ==== 171+0+0 (3492514794 0 0) 0x7f77e8010810 con 0x5640604f3290
2016-06-02 14:09:38.302070 7f7855fa2ac0  1 -- 127.0.0.1:0/2056988180 mark_down 0x5640604f3290 -- 0x5640604f7900
2016-06-02 14:09:38.302154 7f7855fa2ac0  1 -- 127.0.0.1:0/2056988180 mark_down 0x5640604ec9d0 -- 0x5640604ecb20
2016-06-02 14:09:38.302310 7f7855fa2ac0  1 -- 127.0.0.1:0/2056988180 mark_down_all
2016-06-02 14:09:38.302380 7f7855fa2ac0  1 -- 127.0.0.1:0/2056988180 shutdown complete.
2016-06-02 14:09:38.304208 7f7855fa2ac0  0 lockdep stop