2016-10-25 13:59:21.686661 7f7920ff1700 1 ====== starting new request req=0x7f7920fee7e0 ===== 2016-10-25 13:59:21.686671 7f7920ff1700 2 req 1192:0.000009::DELETE /bucket26/::initializing for trans_id = tx0000000000000000004a8-00580f4919-100f-us-west 2016-10-25 13:59:21.686675 7f7920ff1700 10 rgw api priority: s3=5 s3website=4 2016-10-25 13:59:21.686676 7f7920ff1700 10 host=127.0.0.1 2016-10-25 13:59:21.686678 7f7920ff1700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2016-10-25 13:59:21.686679 7f7920ff1700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/bucket26/ 2016-10-25 13:59:21.686693 7f7920ff1700 20 get_handler handler=25RGWHandler_REST_Bucket_S3 2016-10-25 13:59:21.686696 7f7920ff1700 10 handler=25RGWHandler_REST_Bucket_S3 2016-10-25 13:59:21.686696 7f7920ff1700 2 req 1192:0.000035:s3:DELETE /bucket26/::getting op 2 2016-10-25 13:59:21.686698 7f7920ff1700 10 op=27RGWDeleteBucket_ObjStore_S3 2016-10-25 13:59:21.686699 7f7920ff1700 2 req 1192:0.000038:s3:DELETE /bucket26/:delete_bucket:authorizing 2016-10-25 13:59:21.686716 7f7920ff1700 10 get_canon_resource(): dest=/bucket26/ 2016-10-25 13:59:21.686717 7f7920ff1700 10 auth_hdr: DELETE Tue, 25 Oct 2016 11:59:21 GMT /bucket26/ 2016-10-25 13:59:21.686742 7f7920ff1700 15 calculated digest=zI3Xd9rxGfrQdFLfNAJnOFiM0Aw= 2016-10-25 13:59:21.686744 7f7920ff1700 15 auth_sign=zI3Xd9rxGfrQdFLfNAJnOFiM0Aw= 2016-10-25 13:59:21.686744 7f7920ff1700 15 compare=0 2016-10-25 13:59:21.686745 7f7920ff1700 2 req 1192:0.000084:s3:DELETE /bucket26/:delete_bucket:normalizing buckets and tenants 2016-10-25 13:59:21.686747 7f7920ff1700 10 s->object= s->bucket=bucket26 2016-10-25 13:59:21.686748 7f7920ff1700 2 req 1192:0.000087:s3:DELETE /bucket26/:delete_bucket:init permissions 2016-10-25 13:59:21.686763 7f7920ff1700 15 decode_policy Read AccessControlPolicyuser1user1user1user1FULL_CONTROL 2016-10-25 13:59:21.686770 7f7920ff1700 2 req 1192:0.000109:s3:DELETE /bucket26/:delete_bucket:recalculating target 2016-10-25 13:59:21.686772 7f7920ff1700 2 req 1192:0.000111:s3:DELETE /bucket26/:delete_bucket:reading permissions 2016-10-25 13:59:21.686773 7f7920ff1700 2 req 1192:0.000112:s3:DELETE /bucket26/:delete_bucket:init op 2016-10-25 13:59:21.686774 7f7920ff1700 2 req 1192:0.000113:s3:DELETE /bucket26/:delete_bucket:verifying op mask 2016-10-25 13:59:21.686775 7f7920ff1700 20 required_mask= 4 user.op_mask=7 2016-10-25 13:59:21.686775 7f7920ff1700 2 req 1192:0.000114:s3:DELETE /bucket26/:delete_bucket:verifying op permissions 2016-10-25 13:59:21.686777 7f7920ff1700 5 Searching permissions for uid=user1 mask=50 2016-10-25 13:59:21.686778 7f7920ff1700 5 Found permission: 15 2016-10-25 13:59:21.686779 7f7920ff1700 5 Searching permissions for group=1 mask=50 2016-10-25 13:59:21.686780 7f7920ff1700 5 Permissions for group not found 2016-10-25 13:59:21.686780 7f7920ff1700 5 Searching permissions for group=2 mask=50 2016-10-25 13:59:21.686781 7f7920ff1700 5 Permissions for group not found 2016-10-25 13:59:21.686781 7f7920ff1700 5 Getting permissions id=user1 owner=user1 perm=2 2016-10-25 13:59:21.686781 7f7920ff1700 10 uid=user1 requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 2016-10-25 13:59:21.686782 7f7920ff1700 2 req 1192:0.000121:s3:DELETE /bucket26/:delete_bucket:verifying op params 2016-10-25 13:59:21.686783 7f7920ff1700 2 req 1192:0.000122:s3:DELETE /bucket26/:delete_bucket:pre-executing 2016-10-25 13:59:21.686794 7f7920ff1700 2 req 1192:0.000133:s3:DELETE /bucket26/:delete_bucket:executing 2016-10-25 13:59:21.686802 7f7920ff1700 20 reading from us-west.rgw.data.root:.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 2016-10-25 13:59:21.686810 7f7920ff1700 20 get_system_obj_state: rctx=0x7f7920fed470 obj=us-west.rgw.data.root:.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 state=0x7f7970037ea8 s->prefetch_data=0 2016-10-25 13:59:21.686817 7f7920ff1700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 : hit (requested=22, cached=23) 2016-10-25 13:59:21.686822 7f7920ff1700 20 get_system_obj_state: s->obj_tag was set empty 2016-10-25 13:59:21.686825 7f7920ff1700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 : hit (requested=17, cached=23) 2016-10-25 13:59:21.686853 7f7920ff1700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6664 9.20175489 .dir.0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 [call rgw.bucket_list] snapc 0=[] ack+read+known_if_redirected e29) v7 -- ?+0 0x7f7970039760 con 0x564acad41190 2016-10-25 13:59:21.749053 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2941 ==== osd_op_reply(6662 .dir.0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 [call] v29'11 uv11 ack = 0) v7 ==== 169+0+0 (1047751592 0 0) 0x7f7a2000bc40 con 0x564acad41190 2016-10-25 13:59:21.749157 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3220 ==== osd_op_reply(6663 gc.24 [call] v29'11 uv11 ack = 0) v7 ==== 125+0+0 (1351465222 0 0) 0x7f7a18015f00 con 0x564acad44c40 2016-10-25 13:59:21.749194 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3221 ==== osd_op_reply(6663 gc.24 [call] v29'11 uv11 ondisk = 0) v7 ==== 125+0+0 (1128602188 0 0) 0x7f7a18015f00 con 0x564acad44c40 2016-10-25 13:59:21.749267 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2942 ==== osd_op_reply(6662 .dir.0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 [call] v29'11 uv11 ondisk = 0) v7 ==== 169+0+0 (767281378 0 0) 0x7f7a2000bc40 con 0x564acad41190 2016-10-25 13:59:21.749593 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2943 ==== osd_op_reply(6664 .dir.0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 [call] v0'0 uv11 ondisk = 0) v7 ==== 169+0+102 (2507748447 0 1024305959) 0x7f7a2000bc40 con 0x564acad41190 2016-10-25 13:59:21.749913 7f7920ff1700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6665 7.4a894d1e user1.buckets [call user.set_buckets_info] snapc 0=[] ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f7970031610 con 0x564acad41190 2016-10-25 13:59:21.750158 7f79197e2700 20 RGWEnv::set(): HTTP_HOST: localhost:8002 2016-10-25 13:59:21.750162 7f79197e2700 20 RGWEnv::set(): HTTP_ACCEPT: */* 2016-10-25 13:59:21.750164 7f79197e2700 20 RGWEnv::set(): HTTP_TRANSFER_ENCODING: chunked 2016-10-25 13:59:21.750165 7f79197e2700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS 1555b35654ad1656d804:o7KGzMZ4oWajRmNPV5dw2RKuNKQ= 2016-10-25 13:59:21.750167 7f79197e2700 20 RGWEnv::set(): HTTP_DATE: Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.750168 7f79197e2700 20 RGWEnv::set(): HTTP_EXPECT: 100-continue 2016-10-25 13:59:21.750169 7f79197e2700 20 RGWEnv::set(): REQUEST_METHOD: GET 2016-10-25 13:59:21.750170 7f79197e2700 20 RGWEnv::set(): REQUEST_URI: /admin/log 2016-10-25 13:59:21.750171 7f79197e2700 20 RGWEnv::set(): QUERY_STRING: bucket-instance=bucket24%3A0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24&format=json&marker&type=bucket-index&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896 2016-10-25 13:59:21.750173 7f79197e2700 20 RGWEnv::set(): REMOTE_USER: 2016-10-25 13:59:21.750174 7f79197e2700 20 RGWEnv::set(): SCRIPT_URI: /admin/log 2016-10-25 13:59:21.750175 7f79197e2700 20 RGWEnv::set(): SERVER_PORT: 8002 2016-10-25 13:59:21.750175 7f79197e2700 20 HTTP_ACCEPT=*/* 2016-10-25 13:59:21.750176 7f79197e2700 20 HTTP_AUTHORIZATION=AWS 1555b35654ad1656d804:o7KGzMZ4oWajRmNPV5dw2RKuNKQ= 2016-10-25 13:59:21.750176 7f79197e2700 20 HTTP_DATE=Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.750177 7f79197e2700 20 HTTP_EXPECT=100-continue 2016-10-25 13:59:21.750177 7f79197e2700 20 HTTP_HOST=localhost:8002 2016-10-25 13:59:21.750177 7f79197e2700 20 HTTP_TRANSFER_ENCODING=chunked 2016-10-25 13:59:21.750178 7f79197e2700 20 QUERY_STRING=bucket-instance=bucket24%3A0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24&format=json&marker&type=bucket-index&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896 2016-10-25 13:59:21.750178 7f79197e2700 20 REMOTE_USER= 2016-10-25 13:59:21.750179 7f79197e2700 20 REQUEST_METHOD=GET 2016-10-25 13:59:21.750179 7f79197e2700 20 REQUEST_URI=/admin/log 2016-10-25 13:59:21.750179 7f79197e2700 20 SCRIPT_URI=/admin/log 2016-10-25 13:59:21.750180 7f79197e2700 20 SERVER_PORT=8002 2016-10-25 13:59:21.750181 7f79197e2700 1 ====== starting new request req=0x7f79197df7e0 ===== 2016-10-25 13:59:21.750190 7f79197e2700 2 req 1193:0.000009::GET /admin/log::initializing for trans_id = tx0000000000000000004a9-00580f4919-100f-us-west 2016-10-25 13:59:21.750196 7f79197e2700 10 rgw api priority: s3=5 s3website=4 2016-10-25 13:59:21.750196 7f79197e2700 10 host=localhost 2016-10-25 13:59:21.750198 7f79197e2700 20 subdomain= domain=localhost in_hosted_domain=1 in_hosted_domain_s3website=0 2016-10-25 13:59:21.750199 7f79197e2700 20 final domain/bucket subdomain= domain=localhost in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=localhost s->info.request_uri=/admin/log 2016-10-25 13:59:21.750219 7f79197e2700 10 handler=14RGWHandler_Log 2016-10-25 13:59:21.750220 7f79197e2700 2 req 1193:0.000039::GET /admin/log::getting op 0 2016-10-25 13:59:21.750222 7f79197e2700 10 op=16RGWOp_BILog_List 2016-10-25 13:59:21.750222 7f79197e2700 2 req 1193:0.000042::GET /admin/log:list_bucket_index_log:authorizing 2016-10-25 13:59:21.750237 7f79197e2700 10 get_canon_resource(): dest=/admin/log 2016-10-25 13:59:21.750238 7f79197e2700 10 auth_hdr: GET Tue Oct 25 11:59:21 2016 /admin/log 2016-10-25 13:59:21.750259 7f79197e2700 15 calculated digest=o7KGzMZ4oWajRmNPV5dw2RKuNKQ= 2016-10-25 13:59:21.750261 7f79197e2700 15 auth_sign=o7KGzMZ4oWajRmNPV5dw2RKuNKQ= 2016-10-25 13:59:21.750261 7f79197e2700 15 compare=0 2016-10-25 13:59:21.750262 7f79197e2700 20 system request 2016-10-25 13:59:21.750264 7f79197e2700 2 req 1193:0.000083::GET /admin/log:list_bucket_index_log:normalizing buckets and tenants 2016-10-25 13:59:21.750265 7f79197e2700 2 req 1193:0.000084::GET /admin/log:list_bucket_index_log:init permissions 2016-10-25 13:59:21.750267 7f79197e2700 2 req 1193:0.000086::GET /admin/log:list_bucket_index_log:recalculating target 2016-10-25 13:59:21.750267 7f79197e2700 2 req 1193:0.000086::GET /admin/log:list_bucket_index_log:reading permissions 2016-10-25 13:59:21.750268 7f79197e2700 2 req 1193:0.000087::GET /admin/log:list_bucket_index_log:init op 2016-10-25 13:59:21.750269 7f79197e2700 2 req 1193:0.000088::GET /admin/log:list_bucket_index_log:verifying op mask 2016-10-25 13:59:21.750270 7f79197e2700 20 required_mask= 0 user.op_mask=7 2016-10-25 13:59:21.750270 7f79197e2700 2 req 1193:0.000089::GET /admin/log:list_bucket_index_log:verifying op permissions 2016-10-25 13:59:21.750271 7f79197e2700 2 overriding permissions due to system operation 2016-10-25 13:59:21.750272 7f79197e2700 2 req 1193:0.000091::GET /admin/log:list_bucket_index_log:verifying op params 2016-10-25 13:59:21.750272 7f79197e2700 2 req 1193:0.000092::GET /admin/log:list_bucket_index_log:pre-executing 2016-10-25 13:59:21.750273 7f79197e2700 2 req 1193:0.000092::GET /admin/log:list_bucket_index_log:executing 2016-10-25 13:59:21.750276 7f79197e2700 20 reading from us-west.rgw.data.root:.bucket.meta.bucket24:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24 2016-10-25 13:59:21.750283 7f79197e2700 20 get_system_obj_state: rctx=0x7f79197df6f0 obj=us-west.rgw.data.root:.bucket.meta.bucket24:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24 state=0x7f79b002c8f8 s->prefetch_data=0 2016-10-25 13:59:21.750290 7f79197e2700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket24:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24 : hit (requested=22, cached=23) 2016-10-25 13:59:21.750296 7f79197e2700 20 get_system_obj_state: s->obj_tag was set empty 2016-10-25 13:59:21.750299 7f79197e2700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket24:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24 : hit (requested=17, cached=23) 2016-10-25 13:59:21.750318 7f79197e2700 20 list_bi_log_entries: bucket24(@{i=us-west.rgw.buckets.index,e=us-west.rgw.buckets.non-ec}us-west.rgw.buckets.data[0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24]) marker shard_id=-1 max 1000 2016-10-25 13:59:21.750326 7f79197e2700 20 reading from us-west.rgw.data.root:.bucket.meta.bucket24:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24 2016-10-25 13:59:21.750332 7f79197e2700 20 get_system_obj_state: rctx=0x7f79197de4b0 obj=us-west.rgw.data.root:.bucket.meta.bucket24:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24 state=0x7f79b002fe38 s->prefetch_data=0 2016-10-25 13:59:21.750336 7f79197e2700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket24:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24 : hit (requested=22, cached=23) 2016-10-25 13:59:21.750339 7f79197e2700 20 get_system_obj_state: s->obj_tag was set empty 2016-10-25 13:59:21.750341 7f79197e2700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket24:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24 : hit (requested=17, cached=23) 2016-10-25 13:59:21.750382 7f79197e2700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6823/27883 -- osd_op(client.4111.0:6666 9.1a528a56 .dir.0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24 [call rgw.bi_log_list] snapc 0=[] ack+read+known_if_redirected e29) v7 -- ?+0 0x7f79b0030280 con 0x564acad46f70 2016-10-25 13:59:21.750943 7f7a407f8700 1 -- 127.0.0.1:0/1778501402 <== osd.2 127.0.0.1:6823/27883 2208 ==== osd_op_reply(6666 .dir.0474fe09-15db-4e91-8f51-ff360bdeb074.4138.24 [call] v0'0 uv34 ondisk = 0) v7 ==== 169+0+11 (3294495150 0 1993775135) 0x7f7a10019350 con 0x564acad46f70 2016-10-25 13:59:21.751169 7f79197e2700 2 req 1193:0.000988::GET /admin/log:list_bucket_index_log:completing 2016-10-25 13:59:21.751196 7f79197e2700 2 req 1193:0.001015::GET /admin/log:list_bucket_index_log:op status=0 2016-10-25 13:59:21.751198 7f79197e2700 2 req 1193:0.001018::GET /admin/log:list_bucket_index_log:http status=200 2016-10-25 13:59:21.751200 7f79197e2700 1 ====== req done req=0x7f79197df7e0 op status=0 http_status=200 ====== 2016-10-25 13:59:21.751222 7f79197e2700 1 civetweb: 0x7f79b0000c80: 127.0.0.1 - - [25/Oct/2016:13:59:21 +0200] "GET /admin/log HTTP/1.1" 200 0 - - 2016-10-25 13:59:21.767679 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2944 ==== osd_op_reply(6665 user1.buckets [call] v29'83 uv83 ondisk = 0) v7 ==== 133+0+0 (2551587458 0 0) 0x7f7a2000bc40 con 0x564acad41190 2016-10-25 13:59:21.767728 7f7920ff1700 0 sending create_bucket request to master zonegroup 2016-10-25 13:59:21.767753 7f7920ff1700 20 RGWEnv::set(): HTTP_DATE: Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.767756 7f7920ff1700 20 > HTTP_DATE -> Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.767764 7f7920ff1700 10 get_canon_resource(): dest=/bucket26/ 2016-10-25 13:59:21.767766 7f7920ff1700 10 generated canonical header: DELETE Tue Oct 25 11:59:21 2016 /bucket26/ 2016-10-25 13:59:21.767789 7f7920ff1700 15 generated auth header: AWS 1555b35654ad1656d804:rHj+QbdR06xIJ70gnQ0ao4gKGfM= 2016-10-25 13:59:21.767804 7f7920ff1700 20 sending request to http://localhost:8001/bucket26/?&rgwx-uid=user1&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896&rgwx-tag=_hg34y-yn2TIrLw1_4bZ9TAg&rgwx-ver=1 2016-10-25 13:59:21.791092 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f79647e01f0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.791107 7f797d7fa700 20 rgw meta sync: incremental_sync:1542: shard_id=37 mdlog_marker=1_1477396754.819393_235.1 sync_marker.marker=1_1477396754.819393_235.1 period_marker= 2016-10-25 13:59:21.791109 7f797d7fa700 20 rgw meta sync: incremental_sync:1549: shard_id=37 syncing mdlog for shard_id=37 2016-10-25 13:59:21.791143 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.791145 7f797d7fa700 20 rgw meta sync: operate: shard_id=37: init request 2016-10-25 13:59:21.791146 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.791146 7f797d7fa700 20 rgw meta sync: operate: shard_id=37: reading shard status 2016-10-25 13:59:21.791171 7f797d7fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6667 5.8dff6bcb meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.37 [call log.info] snapc 0=[] ack+read+known_if_redirected e29) v7 -- ?+0 0x7f7964c82510 con 0x564acad41190 2016-10-25 13:59:21.791183 7f797d7fa700 20 run: stack=0x7f7964767c20 is io blocked 2016-10-25 13:59:21.791658 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2945 ==== osd_op_reply(6667 meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.37 [call] v0'0 uv197 ondisk = 0) v7 ==== 168+0+49 (3846714337 0 1593169064) 0x7f7a2000bc40 con 0x564acad41190 2016-10-25 13:59:21.791707 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.791711 7f797d7fa700 20 rgw meta sync: operate: shard_id=37: reading shard status complete 2016-10-25 13:59:21.791712 7f797d7fa700 20 rgw meta sync: shard_id=37 marker=1_1477396754.819393_235.1 last_update=2016-10-25 13:59:14.0.819393s 2016-10-25 13:59:21.791719 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.791720 7f797d7fa700 20 rgw meta sync: operate: shard_id=37: sending rest request 2016-10-25 13:59:21.791788 7f797d7fa700 20 RGWEnv::set(): HTTP_DATE: Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.791791 7f797d7fa700 20 > HTTP_DATE -> Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.791795 7f797d7fa700 10 get_canon_resource(): dest=/admin/log 2016-10-25 13:59:21.791796 7f797d7fa700 10 generated canonical header: GET Tue Oct 25 11:59:21 2016 /admin/log 2016-10-25 13:59:21.791821 7f797d7fa700 15 generated auth header: AWS 1555b35654ad1656d804:o7KGzMZ4oWajRmNPV5dw2RKuNKQ= 2016-10-25 13:59:21.791829 7f797d7fa700 20 sending request to http://localhost:8001/admin/log?type=metadata&id=37&period=3cf65d7f-574c-4060-bd40-3332a612b201&max-entries=100&marker=1_1477396754.819393_235.1&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896 2016-10-25 13:59:21.791836 7f797d7fa700 20 register_request mgr=0x564acad64a90 req_data->id=524, easy_handle=0x7f7964e74cd0 2016-10-25 13:59:21.791844 7f797d7fa700 20 run: stack=0x7f7964767c20 is io blocked 2016-10-25 13:59:21.791851 7f797dffb700 20 link_request req_data=0x7f7964e503b0 req_data->id=524, easy_handle=0x7f7964e74cd0 2016-10-25 13:59:21.817868 7f797d7fa700 20 cr:s=0x7f7964011fd0:op=0x7f7964451ef0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.817879 7f797d7fa700 20 rgw meta sync: incremental_sync:1542: shard_id=21 mdlog_marker=1_1477396758.800244_280.1 sync_marker.marker=1_1477396755.217239_278.1 period_marker= 2016-10-25 13:59:21.817883 7f797d7fa700 20 rgw meta sync: incremental_sync:1549: shard_id=21 syncing mdlog for shard_id=21 2016-10-25 13:59:21.817891 7f797d7fa700 20 cr:s=0x7f7964011fd0:op=0x7f7964e50d70:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.817893 7f797d7fa700 20 rgw meta sync: operate: shard_id=21: init request 2016-10-25 13:59:21.817894 7f797d7fa700 20 cr:s=0x7f7964011fd0:op=0x7f7964e50d70:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.817895 7f797d7fa700 20 rgw meta sync: operate: shard_id=21: reading shard status 2016-10-25 13:59:21.817934 7f797d7fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6668 5.20e8338c meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.21 [call log.info] snapc 0=[] ack+read+known_if_redirected e29) v7 -- ?+0 0x7f7964e7fc20 con 0x564acad41190 2016-10-25 13:59:21.817948 7f797d7fa700 20 run: stack=0x7f7964011fd0 is io blocked 2016-10-25 13:59:21.818389 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2946 ==== osd_op_reply(6668 meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.21 [call] v0'0 uv209 ondisk = 0) v7 ==== 168+0+49 (37125735 0 3697331909) 0x7f7a2000bc40 con 0x564acad41190 2016-10-25 13:59:21.818439 7f797d7fa700 20 cr:s=0x7f7964011fd0:op=0x7f7964e50d70:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.818443 7f797d7fa700 20 rgw meta sync: operate: shard_id=21: reading shard status complete 2016-10-25 13:59:21.818444 7f797d7fa700 20 rgw meta sync: shard_id=21 marker=1_1477396758.800244_280.1 last_update=2016-10-25 13:59:18.0.800244s 2016-10-25 13:59:21.818450 7f797d7fa700 20 cr:s=0x7f7964011fd0:op=0x7f7964e50d70:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.818451 7f797d7fa700 20 rgw meta sync: operate: shard_id=21: sending rest request 2016-10-25 13:59:21.818529 7f797d7fa700 20 RGWEnv::set(): HTTP_DATE: Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.818532 7f797d7fa700 20 > HTTP_DATE -> Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.818536 7f797d7fa700 10 get_canon_resource(): dest=/admin/log 2016-10-25 13:59:21.818537 7f797d7fa700 10 generated canonical header: GET Tue Oct 25 11:59:21 2016 /admin/log 2016-10-25 13:59:21.818560 7f797d7fa700 15 generated auth header: AWS 1555b35654ad1656d804:o7KGzMZ4oWajRmNPV5dw2RKuNKQ= 2016-10-25 13:59:21.818573 7f797d7fa700 20 sending request to http://localhost:8001/admin/log?type=metadata&id=21&period=3cf65d7f-574c-4060-bd40-3332a612b201&max-entries=100&marker=1_1477396758.800244_280.1&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896 2016-10-25 13:59:21.818580 7f797d7fa700 20 register_request mgr=0x564acad64a90 req_data->id=525, easy_handle=0x7f7964e95ee0 2016-10-25 13:59:21.818588 7f797d7fa700 20 run: stack=0x7f7964011fd0 is io blocked 2016-10-25 13:59:21.818594 7f797dffb700 20 link_request req_data=0x7f7964e95ce0 req_data->id=525, easy_handle=0x7f7964e95ee0 2016-10-25 13:59:21.819366 7f79067bc700 20 execute(): read data: [53] 2016-10-25 13:59:21.819402 7f79067bc700 20 execute(): updated shard=53 2016-10-25 13:59:21.819414 7f79067bc700 2 req 1174:1.001195::POST /admin/log:mdlog_notify:completing 2016-10-25 13:59:21.819427 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964790dd0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.819432 7f797d7fa700 20 rgw meta sync: incremental_sync:1542: shard_id=53 mdlog_marker=1_1477396753.762765_286.1 sync_marker.marker=1_1477396753.762765_286.1 period_marker= 2016-10-25 13:59:21.819433 7f797d7fa700 20 rgw meta sync: incremental_sync:1549: shard_id=53 syncing mdlog for shard_id=53 2016-10-25 13:59:21.819439 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e9f280:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.819439 7f797d7fa700 20 rgw meta sync: operate: shard_id=53: init request 2016-10-25 13:59:21.819440 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e9f280:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.819441 7f797d7fa700 20 rgw meta sync: operate: shard_id=53: reading shard status 2016-10-25 13:59:21.819455 7f79067bc700 2 req 1174:1.001236::POST /admin/log:mdlog_notify:op status=0 2016-10-25 13:59:21.819459 7f79067bc700 2 req 1174:1.001240::POST /admin/log:mdlog_notify:http status=200 2016-10-25 13:59:21.819462 7f79067bc700 1 ====== req done req=0x7f79067b97e0 op status=0 http_status=200 ====== 2016-10-25 13:59:21.819463 7f797d7fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6669 5.16290def meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.53 [call log.info] snapc 0=[] ack+read+known_if_redirected e29) v7 -- ?+0 0x7f7964ea2540 con 0x564acad41190 2016-10-25 13:59:21.819473 7f797d7fa700 20 run: stack=0x7f79644b91c0 is io blocked 2016-10-25 13:59:21.819491 7f79067bc700 1 civetweb: 0x7f7950004f10: 127.0.0.1 - - [25/Oct/2016:13:59:20 +0200] "POST /admin/log HTTP/1.1" 200 0 - - 2016-10-25 13:59:21.819812 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2947 ==== osd_op_reply(6669 meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.53 [call] v0'0 uv180 ondisk = 0) v7 ==== 168+0+49 (96089988 0 267373447) 0x7f7a2000bc40 con 0x564acad41190 2016-10-25 13:59:21.819843 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e9f280:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.819847 7f797d7fa700 20 rgw meta sync: operate: shard_id=53: reading shard status complete 2016-10-25 13:59:21.819848 7f797d7fa700 20 rgw meta sync: shard_id=53 marker=1_1477396753.762765_286.1 last_update=2016-10-25 13:59:13.0.762765s 2016-10-25 13:59:21.819853 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e9f280:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.819854 7f797d7fa700 20 rgw meta sync: operate: shard_id=53: sending rest request 2016-10-25 13:59:21.819924 7f797d7fa700 20 RGWEnv::set(): HTTP_DATE: Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.819927 7f797d7fa700 20 > HTTP_DATE -> Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.819931 7f797d7fa700 10 get_canon_resource(): dest=/admin/log 2016-10-25 13:59:21.819932 7f797d7fa700 10 generated canonical header: GET Tue Oct 25 11:59:21 2016 /admin/log 2016-10-25 13:59:21.819949 7f797d7fa700 15 generated auth header: AWS 1555b35654ad1656d804:o7KGzMZ4oWajRmNPV5dw2RKuNKQ= 2016-10-25 13:59:21.819961 7f797d7fa700 20 sending request to http://localhost:8001/admin/log?type=metadata&id=53&period=3cf65d7f-574c-4060-bd40-3332a612b201&max-entries=100&marker=1_1477396753.762765_286.1&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896 2016-10-25 13:59:21.819966 7f797d7fa700 20 register_request mgr=0x564acad64a90 req_data->id=526, easy_handle=0x7f7964eb8a10 2016-10-25 13:59:21.819972 7f797d7fa700 20 run: stack=0x7f79644b91c0 is io blocked 2016-10-25 13:59:21.819979 7f797dffb700 20 link_request req_data=0x7f7964eb8810 req_data->id=526, easy_handle=0x7f7964eb8a10 2016-10-25 13:59:21.820231 7f79397fa700 20 RGWEnv::set(): HTTP_HOST: localhost:8002 2016-10-25 13:59:21.820236 7f79397fa700 20 RGWEnv::set(): HTTP_ACCEPT: */* 2016-10-25 13:59:21.820237 7f79397fa700 20 RGWEnv::set(): HTTP_TRANSFER_ENCODING: chunked 2016-10-25 13:59:21.820238 7f79397fa700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS 1555b35654ad1656d804:teMGI+vTmlaCWC3aEFLCyb+WhK4= 2016-10-25 13:59:21.820239 7f79397fa700 20 RGWEnv::set(): HTTP_DATE: Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.820240 7f79397fa700 20 RGWEnv::set(): HTTP_EXPECT: 100-continue 2016-10-25 13:59:21.820241 7f79397fa700 20 RGWEnv::set(): REQUEST_METHOD: POST 2016-10-25 13:59:21.820242 7f79397fa700 20 RGWEnv::set(): REQUEST_URI: /admin/log 2016-10-25 13:59:21.820243 7f79397fa700 20 RGWEnv::set(): QUERY_STRING: type=metadata¬ify&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896 2016-10-25 13:59:21.820244 7f79397fa700 20 RGWEnv::set(): REMOTE_USER: 2016-10-25 13:59:21.820245 7f79397fa700 20 RGWEnv::set(): SCRIPT_URI: /admin/log 2016-10-25 13:59:21.820246 7f79397fa700 20 RGWEnv::set(): SERVER_PORT: 8002 2016-10-25 13:59:21.820250 7f79397fa700 20 HTTP_ACCEPT=*/* 2016-10-25 13:59:21.820251 7f79397fa700 20 HTTP_AUTHORIZATION=AWS 1555b35654ad1656d804:teMGI+vTmlaCWC3aEFLCyb+WhK4= 2016-10-25 13:59:21.820254 7f79397fa700 20 HTTP_DATE=Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.820254 7f79397fa700 20 HTTP_EXPECT=100-continue 2016-10-25 13:59:21.820255 7f79397fa700 20 HTTP_HOST=localhost:8002 2016-10-25 13:59:21.820255 7f79397fa700 20 HTTP_TRANSFER_ENCODING=chunked 2016-10-25 13:59:21.820256 7f79397fa700 20 QUERY_STRING=type=metadata¬ify&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896 2016-10-25 13:59:21.820256 7f79397fa700 20 REMOTE_USER= 2016-10-25 13:59:21.820257 7f79397fa700 20 REQUEST_METHOD=POST 2016-10-25 13:59:21.820257 7f79397fa700 20 REQUEST_URI=/admin/log 2016-10-25 13:59:21.820257 7f79397fa700 20 SCRIPT_URI=/admin/log 2016-10-25 13:59:21.820258 7f79397fa700 20 SERVER_PORT=8002 2016-10-25 13:59:21.820259 7f79397fa700 1 ====== starting new request req=0x7f79397f77e0 ===== 2016-10-25 13:59:21.820268 7f79397fa700 2 req 1194:0.000008::POST /admin/log::initializing for trans_id = tx0000000000000000004aa-00580f4919-100f-us-west 2016-10-25 13:59:21.820272 7f79397fa700 10 rgw api priority: s3=5 s3website=4 2016-10-25 13:59:21.820273 7f79397fa700 10 host=localhost 2016-10-25 13:59:21.820274 7f79397fa700 20 subdomain= domain=localhost in_hosted_domain=1 in_hosted_domain_s3website=0 2016-10-25 13:59:21.820275 7f79397fa700 20 final domain/bucket subdomain= domain=localhost in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=localhost s->info.request_uri=/admin/log 2016-10-25 13:59:21.820290 7f79397fa700 10 handler=14RGWHandler_Log 2016-10-25 13:59:21.820291 7f79397fa700 2 req 1194:0.000032::POST /admin/log::getting op 4 2016-10-25 13:59:21.820293 7f79397fa700 10 op=18RGWOp_MDLog_Notify 2016-10-25 13:59:21.820294 7f79397fa700 2 req 1194:0.000035::POST /admin/log:mdlog_notify:authorizing 2016-10-25 13:59:21.820305 7f79397fa700 10 get_canon_resource(): dest=/admin/log 2016-10-25 13:59:21.820306 7f79397fa700 10 auth_hdr: POST Tue Oct 25 11:59:21 2016 /admin/log 2016-10-25 13:59:21.820321 7f79397fa700 15 calculated digest=teMGI+vTmlaCWC3aEFLCyb+WhK4= 2016-10-25 13:59:21.820324 7f79397fa700 15 auth_sign=teMGI+vTmlaCWC3aEFLCyb+WhK4= 2016-10-25 13:59:21.820324 7f79397fa700 15 compare=0 2016-10-25 13:59:21.820325 7f79397fa700 20 system request 2016-10-25 13:59:21.820327 7f79397fa700 2 req 1194:0.000068::POST /admin/log:mdlog_notify:normalizing buckets and tenants 2016-10-25 13:59:21.820328 7f79397fa700 2 req 1194:0.000069::POST /admin/log:mdlog_notify:init permissions 2016-10-25 13:59:21.820329 7f79397fa700 2 req 1194:0.000070::POST /admin/log:mdlog_notify:recalculating target 2016-10-25 13:59:21.820330 7f79397fa700 2 req 1194:0.000071::POST /admin/log:mdlog_notify:reading permissions 2016-10-25 13:59:21.820331 7f79397fa700 2 req 1194:0.000072::POST /admin/log:mdlog_notify:init op 2016-10-25 13:59:21.820332 7f79397fa700 2 req 1194:0.000072::POST /admin/log:mdlog_notify:verifying op mask 2016-10-25 13:59:21.820332 7f79397fa700 20 required_mask= 0 user.op_mask=7 2016-10-25 13:59:21.820333 7f79397fa700 2 req 1194:0.000074::POST /admin/log:mdlog_notify:verifying op permissions 2016-10-25 13:59:21.820334 7f79397fa700 2 overriding permissions due to system operation 2016-10-25 13:59:21.820334 7f79397fa700 2 req 1194:0.000075::POST /admin/log:mdlog_notify:verifying op params 2016-10-25 13:59:21.820335 7f79397fa700 2 req 1194:0.000076::POST /admin/log:mdlog_notify:pre-executing 2016-10-25 13:59:21.820336 7f79397fa700 2 req 1194:0.000077::POST /admin/log:mdlog_notify:executing 2016-10-25 13:59:21.844993 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.845004 7f797dffb700 10 received header:HTTP/1.1 200 OK 2016-10-25 13:59:21.845008 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.845009 7f797dffb700 10 received header:x-amz-request-id: tx0000000000000000004a8-00580f4919-102a-us-east-1 2016-10-25 13:59:21.845015 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.845016 7f797dffb700 10 received header:Content-Length: 44 2016-10-25 13:59:21.845019 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.845019 7f797dffb700 10 received header:Date: Tue, 25 Oct 2016 11:59:21 GMT 2016-10-25 13:59:21.845021 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.845022 7f797dffb700 10 received header: 2016-10-25 13:59:21.845061 7f797d7fa700 20 cr:s=0x7f7964011fd0:op=0x7f7964e50d70:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.845065 7f797d7fa700 20 rgw meta sync: operate: shard_id=21: receiving rest response 2016-10-25 13:59:21.845214 7f797d7fa700 20 rgw meta sync: remote mdlog, shard_id=21 num of shard entries: 0 2016-10-25 13:59:21.845222 7f797d7fa700 20 cr:s=0x7f7964011fd0:op=0x7f7964451ef0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.845224 7f797d7fa700 20 rgw meta sync: incremental_sync:1561: shard_id=21 mdlog_marker=1_1477396758.800244_280.1 sync_marker.marker=1_1477396755.217239_278.1 2016-10-25 13:59:21.845228 7f797d7fa700 0 rgw meta sync: cr:s=0x7f7964011fd0:op=0x7f7964451ef0:18RGWMetaSyncShardCR: adjusting marker pos=1_1477396758.738776_279.1 2016-10-25 13:59:21.845230 7f797d7fa700 0 rgw meta sync: cr:s=0x7f7964011fd0:op=0x7f7964451ef0:18RGWMetaSyncShardCR: adjusting marker pos=1_1477396758.800244_280.1 2016-10-25 13:59:21.845231 7f797d7fa700 20 rgw meta sync: incremental_sync:1594: shard_id=21 mdlog_marker=1_1477396758.800244_280.1 max_marker=1_1477396758.800244_280.1 sync_marker.marker=1_1477396758.800244_280.1 period_marker= 2016-10-25 13:59:21.845239 7f797d7fa700 20 run: stack=0x7f7964011fd0 is io blocked 2016-10-25 13:59:21.845374 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.845378 7f797dffb700 10 received header:HTTP/1.1 200 OK 2016-10-25 13:59:21.845380 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.845381 7f797dffb700 10 received header:x-amz-request-id: tx0000000000000000004a9-00580f4919-102a-us-east-1 2016-10-25 13:59:21.845385 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.845385 7f797dffb700 10 received header:Content-Length: 1716 2016-10-25 13:59:21.845388 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.845388 7f797dffb700 10 received header:Date: Tue, 25 Oct 2016 11:59:21 GMT 2016-10-25 13:59:21.845390 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.845391 7f797dffb700 10 received header: 2016-10-25 13:59:21.845416 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e9f280:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.845419 7f797d7fa700 20 rgw meta sync: operate: shard_id=53: receiving rest response 2016-10-25 13:59:21.846008 7f797d7fa700 20 rgw meta sync: remote mdlog, shard_id=53 num of shard entries: 6 2016-10-25 13:59:21.846015 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e9f280:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.846017 7f797d7fa700 20 rgw meta sync: operate: shard_id=53: storing mdlog entries 2016-10-25 13:59:21.846018 7f797d7fa700 20 rgw meta sync: entry: name=bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 2016-10-25 13:59:21.846067 7f797d7fa700 20 rgw meta sync: entry: name=bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 2016-10-25 13:59:21.846072 7f797d7fa700 20 rgw meta sync: entry: name=bucket25 2016-10-25 13:59:21.846074 7f797d7fa700 20 rgw meta sync: entry: name=bucket25 2016-10-25 13:59:21.846076 7f797d7fa700 20 rgw meta sync: entry: name=bucket25 2016-10-25 13:59:21.846078 7f797d7fa700 20 rgw meta sync: entry: name=bucket25 2016-10-25 13:59:21.846123 7f797d7fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6670 5.16290def meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.53 [call log.add] snapc 0=[] ack+ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f7964ea2390 con 0x564acad41190 2016-10-25 13:59:21.846142 7f797d7fa700 20 run: stack=0x7f79644b91c0 is io blocked 2016-10-25 13:59:21.846480 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.846484 7f797dffb700 10 received header:HTTP/1.1 200 OK 2016-10-25 13:59:21.846485 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.846486 7f797dffb700 10 received header:x-amz-request-id: tx0000000000000000004a7-00580f4919-102a-us-east-1 2016-10-25 13:59:21.846488 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.846489 7f797dffb700 10 received header:Content-Length: 1442 2016-10-25 13:59:21.846490 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.846491 7f797dffb700 10 received header:Date: Tue, 25 Oct 2016 11:59:21 GMT 2016-10-25 13:59:21.846492 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.846493 7f797dffb700 10 received header: 2016-10-25 13:59:21.846512 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.846514 7f797d7fa700 20 rgw meta sync: operate: shard_id=37: receiving rest response 2016-10-25 13:59:21.846950 7f797d7fa700 20 rgw meta sync: remote mdlog, shard_id=37 num of shard entries: 5 2016-10-25 13:59:21.846957 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.846960 7f797d7fa700 20 rgw meta sync: operate: shard_id=37: storing mdlog entries 2016-10-25 13:59:21.846961 7f797d7fa700 20 rgw meta sync: entry: name=bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 2016-10-25 13:59:21.847022 7f797d7fa700 20 rgw meta sync: entry: name=bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 2016-10-25 13:59:21.847027 7f797d7fa700 20 rgw meta sync: entry: name=bucket26 2016-10-25 13:59:21.847029 7f797d7fa700 20 rgw meta sync: entry: name=bucket26 2016-10-25 13:59:21.847031 7f797d7fa700 20 rgw meta sync: entry: name=bucket26 2016-10-25 13:59:21.847070 7f797d7fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6671 5.8dff6bcb meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.37 [call log.add] snapc 0=[] ack+ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f7964c82510 con 0x564acad41190 2016-10-25 13:59:21.847087 7f797d7fa700 20 run: stack=0x7f7964767c20 is io blocked 2016-10-25 13:59:21.869186 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2948 ==== osd_op_reply(6670 meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.53 [call] v29'181 uv181 ack = 0) v7 ==== 168+0+0 (1769141992 0 0) 0x7f7a2000bc40 con 0x564acad41190 2016-10-25 13:59:21.869276 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2949 ==== osd_op_reply(6670 meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.53 [call] v29'181 uv181 ondisk = 0) v7 ==== 168+0+0 (2059031970 0 0) 0x7f7a2000bc40 con 0x564acad41190 2016-10-25 13:59:21.869433 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e9f280:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.869440 7f797d7fa700 20 rgw meta sync: operate: shard_id=53: storing mdlog entries complete 2016-10-25 13:59:21.869453 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964790dd0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.869455 7f797d7fa700 20 rgw meta sync: incremental_sync:1561: shard_id=53 mdlog_marker=1_1477396760.750357_295.1 sync_marker.marker=1_1477396753.762765_286.1 2016-10-25 13:59:21.869470 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e4cd10:21RGWReadMDLogEntriesCR: operate() 2016-10-25 13:59:21.869472 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e4cd10:21RGWReadMDLogEntriesCR: operate() 2016-10-25 13:59:21.869482 7f797d7fa700 20 enqueued request req=0x7f7964de1bb0 2016-10-25 13:59:21.869482 7f797d7fa700 20 RGWWQ: 2016-10-25 13:59:21.869483 7f797d7fa700 20 req: 0x7f7964de1bb0 2016-10-25 13:59:21.869486 7f797d7fa700 20 run: stack=0x7f79644b91c0 is io blocked 2016-10-25 13:59:21.869492 7f79c0ff9700 20 dequeued request req=0x7f7964de1bb0 2016-10-25 13:59:21.869495 7f79c0ff9700 20 RGWWQ: empty 2016-10-25 13:59:21.869532 7f79c0ff9700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6672 5.16290def meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.53 [call log.list] snapc 0=[] ack+read+known_if_redirected e29) v7 -- ?+0 0x7f799800ff20 con 0x564acad41190 2016-10-25 13:59:21.870221 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2950 ==== osd_op_reply(6672 meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.53 [call] v0'0 uv181 ondisk = 0) v7 ==== 168+0+1030 (1671514455 0 1474111349) 0x7f7a2002a620 con 0x564acad41190 2016-10-25 13:59:21.870271 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e4cd10:21RGWReadMDLogEntriesCR: operate() 2016-10-25 13:59:21.870276 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e4cd10:21RGWReadMDLogEntriesCR: operate() 2016-10-25 13:59:21.870278 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e4cd10:21RGWReadMDLogEntriesCR: operate() 2016-10-25 13:59:21.870280 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964e4cd10:21RGWReadMDLogEntriesCR: operate() 2016-10-25 13:59:21.870282 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964790dd0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.870287 7f797d7fa700 20 rgw meta sync: incremental_sync:1576: shard_id=53 log_entry: 1_1477396759.882221_289.1:bucket.instance:bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25:2016-10-25 13:59:19.882221 2016-10-25 13:59:21.870304 7f797d7fa700 20 cr:s=0x7f7964e7e230:op=0x7f7964de3cb0:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.870307 7f797d7fa700 20 rgw meta sync: skipping pending operation 2016-10-25 13:59:21.870309 7f797d7fa700 20 rgw meta sync: store_marker(): updating marker marker_oid=mdlog.sync-status.shard.53 marker=1_1477396759.882221_289.1 2016-10-25 13:59:21.870319 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964790dd0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.870322 7f797d7fa700 20 rgw meta sync: incremental_sync:1576: shard_id=53 log_entry: 1_1477396759.936034_290.1:bucket.instance:bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25:2016-10-25 13:59:19.936034 2016-10-25 13:59:21.870332 7f797d7fa700 20 cr:s=0x7f7964e7e230:op=0x7f7964e9f280:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.870334 7f797d7fa700 20 cr:s=0x7f7964e501a0:op=0x7f7964de58e0:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.870337 7f797d7fa700 20 rgw meta sync: fetching remote metadata: bucket.instance:bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 2016-10-25 13:59:21.870344 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964790dd0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.870346 7f797d7fa700 20 rgw meta sync: incremental_sync:1576: shard_id=53 log_entry: 1_1477396759.953869_291.1:bucket:bucket25:2016-10-25 13:59:19.953869 2016-10-25 13:59:21.870354 7f797d7fa700 20 cr:s=0x7f7964e7e230:op=0x7f7964e9f280:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.870372 7f797d7fa700 20 enqueued request req=0x7f7964e50740 2016-10-25 13:59:21.870375 7f797d7fa700 20 RGWWQ: 2016-10-25 13:59:21.870375 7f797d7fa700 20 req: 0x7f7964e50740 2016-10-25 13:59:21.870379 7f797d7fa700 20 run: stack=0x7f7964e7e230 is io blocked 2016-10-25 13:59:21.870381 7f797d7fa700 20 cr:s=0x7f7964e501a0:op=0x7f7964eb98d0:23RGWReadRemoteMetadataCR: operate() 2016-10-25 13:59:21.870386 7f799ffff700 20 dequeued request req=0x7f7964e50740 2016-10-25 13:59:21.870389 7f799ffff700 20 RGWWQ: empty 2016-10-25 13:59:21.870412 7f799ffff700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6673 5.b22d8eda mdlog.sync-status.shard.53 [writefull 0~65] snapc 0=[] ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f7990021320 con 0x564acad41190 2016-10-25 13:59:21.870463 7f797d7fa700 20 RGWEnv::set(): HTTP_DATE: Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.870467 7f797d7fa700 20 > HTTP_DATE -> Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.870475 7f797d7fa700 10 get_canon_resource(): dest=/admin/metadata/bucket.instance/bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 2016-10-25 13:59:21.870477 7f797d7fa700 10 generated canonical header: GET Tue Oct 25 11:59:21 2016 /admin/metadata/bucket.instance/bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 2016-10-25 13:59:21.870506 7f797d7fa700 15 generated auth header: AWS 1555b35654ad1656d804:Vjd/TnkOAw2r6gr6+KKEdR9d6Cs= 2016-10-25 13:59:21.870518 7f797d7fa700 20 sending request to http://localhost:8001/admin/metadata/bucket.instance/bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25?key=bucket25%3A0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896 2016-10-25 13:59:21.870526 7f797d7fa700 20 register_request mgr=0x564acad64a90 req_data->id=527, easy_handle=0x7f7964e6bac0 2016-10-25 13:59:21.870535 7f797d7fa700 20 run: stack=0x7f7964e501a0 is io blocked 2016-10-25 13:59:21.870537 7f797d7fa700 20 cr:s=0x7f7964c1dff0:op=0x7f7964e7f280:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.870539 7f797d7fa700 20 rgw meta sync: skipping pending operation 2016-10-25 13:59:21.870541 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964790dd0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.870544 7f797d7fa700 20 rgw meta sync: incremental_sync:1576: shard_id=53 log_entry: 1_1477396760.006199_292.1:bucket:bucket25:2016-10-25 13:59:20.006199 2016-10-25 13:59:21.870555 7f797d7fa700 20 cr:s=0x7f7964c1dff0:op=0x7f7964e7f280:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.870558 7f797d7fa700 20 run: stack=0x7f7964c1dff0 is done 2016-10-25 13:59:21.870560 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964ebba90:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.870561 7f797d7fa700 20 rgw meta sync: fetching remote metadata: bucket:bucket25 2016-10-25 13:59:21.870566 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964790dd0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.870568 7f797d7fa700 20 rgw meta sync: incremental_sync:1576: shard_id=53 log_entry: 1_1477396760.691960_294.1:bucket:bucket25:2016-10-25 13:59:20.691960 2016-10-25 13:59:21.870576 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964e7f280:23RGWReadRemoteMetadataCR: operate() 2016-10-25 13:59:21.870650 7f797d7fa700 20 RGWEnv::set(): HTTP_DATE: Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.870652 7f797d7fa700 20 > HTTP_DATE -> Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.870657 7f797d7fa700 10 get_canon_resource(): dest=/admin/metadata/bucket/bucket25 2016-10-25 13:59:21.870658 7f797d7fa700 10 generated canonical header: GET Tue Oct 25 11:59:21 2016 /admin/metadata/bucket/bucket25 2016-10-25 13:59:21.870679 7f797d7fa700 15 generated auth header: AWS 1555b35654ad1656d804:UapmCoPP7pXIKJ2tEwfe2/cWJfw= 2016-10-25 13:59:21.870687 7f797d7fa700 20 sending request to http://localhost:8001/admin/metadata/bucket/bucket25?key=bucket25&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896 2016-10-25 13:59:21.870693 7f797d7fa700 20 register_request mgr=0x564acad64a90 req_data->id=528, easy_handle=0x7f7964ea2c50 2016-10-25 13:59:21.870698 7f797d7fa700 20 run: stack=0x7f7964ebc2c0 is io blocked 2016-10-25 13:59:21.870700 7f797d7fa700 20 cr:s=0x7f7964ebd080:op=0x7f7964ebc850:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.870701 7f797d7fa700 20 rgw meta sync: skipping pending operation 2016-10-25 13:59:21.870704 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964790dd0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.870707 7f797d7fa700 20 rgw meta sync: incremental_sync:1576: shard_id=53 log_entry: 1_1477396760.750357_295.1:bucket:bucket25:2016-10-25 13:59:20.750357 2016-10-25 13:59:21.870717 7f797d7fa700 20 cr:s=0x7f7964ebd080:op=0x7f7964ebc850:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.870719 7f797d7fa700 20 run: stack=0x7f7964ebd080 is done 2016-10-25 13:59:21.870721 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964e89500:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.870723 7f797d7fa700 20 rgw meta sync: fetching remote metadata: bucket:bucket25 2016-10-25 13:59:21.870728 7f797d7fa700 20 cr:s=0x7f79644b91c0:op=0x7f7964790dd0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.870731 7f797d7fa700 0 rgw meta sync: cr:s=0x7f79644b91c0:op=0x7f7964790dd0:18RGWMetaSyncShardCR: adjusting marker pos=1_1477396753.762765_286.1 2016-10-25 13:59:21.870733 7f797d7fa700 0 rgw meta sync: cr:s=0x7f79644b91c0:op=0x7f7964790dd0:18RGWMetaSyncShardCR: adjusting marker pos=1_1477396753.762765_286.1 2016-10-25 13:59:21.870734 7f797d7fa700 20 rgw meta sync: incremental_sync:1594: shard_id=53 mdlog_marker=1_1477396760.750357_295.1 max_marker=1_1477396760.750357_295.1 sync_marker.marker=1_1477396753.762765_286.1 period_marker= 2016-10-25 13:59:21.870742 7f797d7fa700 20 run: stack=0x7f79644b91c0 is io blocked 2016-10-25 13:59:21.870744 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964ebc960:23RGWReadRemoteMetadataCR: operate() 2016-10-25 13:59:21.870821 7f797d7fa700 20 RGWEnv::set(): HTTP_DATE: Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.870823 7f797d7fa700 20 > HTTP_DATE -> Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.870831 7f797d7fa700 10 get_canon_resource(): dest=/admin/metadata/bucket/bucket25 2016-10-25 13:59:21.870832 7f797d7fa700 10 generated canonical header: GET Tue Oct 25 11:59:21 2016 /admin/metadata/bucket/bucket25 2016-10-25 13:59:21.870850 7f797d7fa700 15 generated auth header: AWS 1555b35654ad1656d804:UapmCoPP7pXIKJ2tEwfe2/cWJfw= 2016-10-25 13:59:21.870859 7f797d7fa700 20 sending request to http://localhost:8001/admin/metadata/bucket/bucket25?key=bucket25&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896 2016-10-25 13:59:21.870865 7f797d7fa700 20 register_request mgr=0x564acad64a90 req_data->id=529, easy_handle=0x7f7964ecd6c0 2016-10-25 13:59:21.870870 7f797d7fa700 20 run: stack=0x7f7964e89d30 is io blocked 2016-10-25 13:59:21.870881 7f797dffb700 20 link_request req_data=0x7f7964ebafb0 req_data->id=527, easy_handle=0x7f7964e6bac0 2016-10-25 13:59:21.870884 7f797dffb700 20 link_request req_data=0x7f7964e94c40 req_data->id=528, easy_handle=0x7f7964ea2c50 2016-10-25 13:59:21.870885 7f797dffb700 20 link_request req_data=0x7f7964ecd4c0 req_data->id=529, easy_handle=0x7f7964ecd6c0 2016-10-25 13:59:21.871273 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871276 7f797dffb700 10 received header:HTTP/1.1 200 OK 2016-10-25 13:59:21.871278 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871279 7f797dffb700 10 received header:x-amz-request-id: tx0000000000000000004aa-00580f4919-102a-us-east-1 2016-10-25 13:59:21.871283 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871284 7f797dffb700 10 received header:Content-Type: application/json 2016-10-25 13:59:21.871287 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871288 7f797dffb700 10 received header:Content-Length: 1123 2016-10-25 13:59:21.871290 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871291 7f797dffb700 10 received header:Date: Tue, 25 Oct 2016 11:59:21 GMT 2016-10-25 13:59:21.871293 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871293 7f797dffb700 10 received header: 2016-10-25 13:59:21.871628 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871632 7f797dffb700 10 received header:HTTP/1.1 404 Not Found 2016-10-25 13:59:21.871635 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871635 7f797dffb700 10 received header:x-amz-request-id: tx0000000000000000004ac-00580f4919-102a-us-east-1 2016-10-25 13:59:21.871639 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871640 7f797dffb700 10 received header:Content-Length: 113 2016-10-25 13:59:21.871642 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871643 7f797dffb700 10 received header:Accept-Ranges: bytes 2016-10-25 13:59:21.871646 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871646 7f797dffb700 10 received header:Content-Type: application/json 2016-10-25 13:59:21.871648 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871649 7f797dffb700 10 received header:Date: Tue, 25 Oct 2016 11:59:21 GMT 2016-10-25 13:59:21.871651 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871651 7f797dffb700 10 received header: 2016-10-25 13:59:21.871812 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871814 7f797dffb700 10 received header:HTTP/1.1 404 Not Found 2016-10-25 13:59:21.871816 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871816 7f797dffb700 10 received header:x-amz-request-id: tx0000000000000000004ab-00580f4919-102a-us-east-1 2016-10-25 13:59:21.871819 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871819 7f797dffb700 10 received header:Content-Length: 113 2016-10-25 13:59:21.871821 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871821 7f797dffb700 10 received header:Accept-Ranges: bytes 2016-10-25 13:59:21.871823 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871823 7f797dffb700 10 received header:Content-Type: application/json 2016-10-25 13:59:21.871825 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871825 7f797dffb700 10 received header:Date: Tue, 25 Oct 2016 11:59:21 GMT 2016-10-25 13:59:21.871827 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.871827 7f797dffb700 10 received header: 2016-10-25 13:59:21.893644 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2951 ==== osd_op_reply(6671 meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.37 [call] v29'198 uv198 ondisk = 0) v7 ==== 168+0+0 (2936472794 0 0) 0x7f7a2002a620 con 0x564acad41190 2016-10-25 13:59:21.893702 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:24RGWCloneMetaLogCoroutine: operate() 2016-10-25 13:59:21.893708 7f797d7fa700 20 rgw meta sync: operate: shard_id=37: storing mdlog entries complete 2016-10-25 13:59:21.893717 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f79647e01f0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.893718 7f797d7fa700 20 rgw meta sync: incremental_sync:1561: shard_id=37 mdlog_marker=1_1477396761.789001_270.1 sync_marker.marker=1_1477396754.819393_235.1 2016-10-25 13:59:21.893729 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:21RGWReadMDLogEntriesCR: operate() 2016-10-25 13:59:21.893731 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:21RGWReadMDLogEntriesCR: operate() 2016-10-25 13:59:21.893739 7f797d7fa700 20 enqueued request req=0x7f7964c53d40 2016-10-25 13:59:21.893740 7f797d7fa700 20 RGWWQ: 2016-10-25 13:59:21.893741 7f797d7fa700 20 req: 0x7f7964c53d40 2016-10-25 13:59:21.893744 7f797d7fa700 20 run: stack=0x7f7964767c20 is io blocked 2016-10-25 13:59:21.893762 7f79e7fff700 20 dequeued request req=0x7f7964c53d40 2016-10-25 13:59:21.893768 7f79e7fff700 20 RGWWQ: empty 2016-10-25 13:59:21.893804 7f79e7fff700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6674 5.8dff6bcb meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.37 [call log.list] snapc 0=[] ack+read+known_if_redirected e29) v7 -- ?+0 0x7f79d40179a0 con 0x564acad41190 2016-10-25 13:59:21.894246 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2952 ==== osd_op_reply(6674 meta.log.3cf65d7f-574c-4060-bd40-3332a612b201.37 [call] v0'0 uv198 ondisk = 0) v7 ==== 168+0+867 (900340876 0 466406956) 0x7f7a20013ba0 con 0x564acad41190 2016-10-25 13:59:21.894298 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:21RGWReadMDLogEntriesCR: operate() 2016-10-25 13:59:21.894302 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:21RGWReadMDLogEntriesCR: operate() 2016-10-25 13:59:21.894303 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:21RGWReadMDLogEntriesCR: operate() 2016-10-25 13:59:21.894304 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f7964cc56e0:21RGWReadMDLogEntriesCR: operate() 2016-10-25 13:59:21.894306 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f79647e01f0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.894308 7f797d7fa700 20 rgw meta sync: incremental_sync:1576: shard_id=37 log_entry: 1_1477396760.906943_265.1:bucket.instance:bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26:2016-10-25 13:59:20.906943 2016-10-25 13:59:21.894321 7f797d7fa700 20 cr:s=0x7f7964e4d5d0:op=0x7f7964e99670:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.894323 7f797d7fa700 20 rgw meta sync: skipping pending operation 2016-10-25 13:59:21.894325 7f797d7fa700 20 rgw meta sync: store_marker(): updating marker marker_oid=mdlog.sync-status.shard.37 marker=1_1477396760.906943_265.1 2016-10-25 13:59:21.894331 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f79647e01f0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.894332 7f797d7fa700 20 rgw meta sync: incremental_sync:1576: shard_id=37 log_entry: 1_1477396760.972302_266.1:bucket.instance:bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26:2016-10-25 13:59:20.972302 2016-10-25 13:59:21.894337 7f797d7fa700 20 cr:s=0x7f7964e4d5d0:op=0x7f7964e4d9b0:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.894340 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ebe500:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.894341 7f797d7fa700 20 rgw meta sync: fetching remote metadata: bucket.instance:bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 2016-10-25 13:59:21.894345 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f79647e01f0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.894346 7f797d7fa700 20 rgw meta sync: incremental_sync:1576: shard_id=37 log_entry: 1_1477396761.006784_267.1:bucket:bucket26:2016-10-25 13:59:21.006784 2016-10-25 13:59:21.894351 7f797d7fa700 20 cr:s=0x7f7964e4d5d0:op=0x7f7964e4d9b0:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.894358 7f797d7fa700 20 enqueued request req=0x7f7964e4aee0 2016-10-25 13:59:21.894360 7f797d7fa700 20 RGWWQ: 2016-10-25 13:59:21.894360 7f797d7fa700 20 req: 0x7f7964e4aee0 2016-10-25 13:59:21.894370 7f797d7fa700 20 run: stack=0x7f7964e4d5d0 is io blocked 2016-10-25 13:59:21.894372 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ebf130:23RGWReadRemoteMetadataCR: operate() 2016-10-25 13:59:21.894389 7f79e6ffd700 20 dequeued request req=0x7f7964e4aee0 2016-10-25 13:59:21.894395 7f79e6ffd700 20 RGWWQ: empty 2016-10-25 13:59:21.894415 7f79e6ffd700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6675 5.2d1ff4fd mdlog.sync-status.shard.37 [writefull 0~65] snapc 0=[] ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f79d0008b60 con 0x564acad41190 2016-10-25 13:59:21.894449 7f797d7fa700 20 RGWEnv::set(): HTTP_DATE: Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.894453 7f797d7fa700 20 > HTTP_DATE -> Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.894458 7f797d7fa700 10 get_canon_resource(): dest=/admin/metadata/bucket.instance/bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 2016-10-25 13:59:21.894461 7f797d7fa700 10 generated canonical header: GET Tue Oct 25 11:59:21 2016 /admin/metadata/bucket.instance/bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 2016-10-25 13:59:21.894487 7f797d7fa700 15 generated auth header: AWS 1555b35654ad1656d804:sO1HMfUZrt6Q6HgD3QlLjElt7kU= 2016-10-25 13:59:21.894495 7f797d7fa700 20 sending request to http://localhost:8001/admin/metadata/bucket.instance/bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26?key=bucket26%3A0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896 2016-10-25 13:59:21.894501 7f797d7fa700 20 register_request mgr=0x564acad64a90 req_data->id=530, easy_handle=0x7f7964eed2f0 2016-10-25 13:59:21.894510 7f797d7fa700 20 run: stack=0x7f7964ebed30 is io blocked 2016-10-25 13:59:21.894513 7f797d7fa700 20 cr:s=0x7f7964ec0370:op=0x7f7964ebfb40:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.894514 7f797d7fa700 20 rgw meta sync: skipping pending operation 2016-10-25 13:59:21.894517 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f79647e01f0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.894519 7f797d7fa700 20 rgw meta sync: incremental_sync:1576: shard_id=37 log_entry: 1_1477396761.066520_268.1:bucket:bucket26:2016-10-25 13:59:21.066520 2016-10-25 13:59:21.894523 7f797dffb700 20 link_request req_data=0x7f7964eed0f0 req_data->id=530, easy_handle=0x7f7964eed2f0 2016-10-25 13:59:21.894533 7f797d7fa700 20 cr:s=0x7f7964ec0370:op=0x7f7964ebfb40:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.894537 7f797d7fa700 20 run: stack=0x7f7964ec0370 is done 2016-10-25 13:59:21.894539 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef69e0:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.894540 7f797d7fa700 20 rgw meta sync: fetching remote metadata: bucket:bucket26 2016-10-25 13:59:21.894545 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f79647e01f0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.894547 7f797d7fa700 20 rgw meta sync: incremental_sync:1576: shard_id=37 log_entry: 1_1477396761.789001_270.1:bucket:bucket26:2016-10-25 13:59:21.789001 2016-10-25 13:59:21.894558 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef7310:23RGWReadRemoteMetadataCR: operate() 2016-10-25 13:59:21.894633 7f797d7fa700 20 RGWEnv::set(): HTTP_DATE: Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.894636 7f797d7fa700 20 > HTTP_DATE -> Tue Oct 25 11:59:21 2016 2016-10-25 13:59:21.894642 7f797d7fa700 10 get_canon_resource(): dest=/admin/metadata/bucket/bucket26 2016-10-25 13:59:21.894646 7f797d7fa700 10 generated canonical header: GET Tue Oct 25 11:59:21 2016 /admin/metadata/bucket/bucket26 2016-10-25 13:59:21.894667 7f797d7fa700 15 generated auth header: AWS 1555b35654ad1656d804:qap3dH+A/GVQyTHYAUHexXgP0IA= 2016-10-25 13:59:21.894677 7f797d7fa700 20 sending request to http://localhost:8001/admin/metadata/bucket/bucket26?key=bucket26&rgwx-zonegroup=5f55ef7d-26a2-46b1-a2bf-b95d73c78896 2016-10-25 13:59:21.894684 7f797d7fa700 20 register_request mgr=0x564acad64a90 req_data->id=531, easy_handle=0x7f7964f104d0 2016-10-25 13:59:21.894690 7f797d7fa700 20 run: stack=0x7f7964eec690 is io blocked 2016-10-25 13:59:21.894692 7f797d7fa700 20 cr:s=0x7f7964ef80e0:op=0x7f7964ef78b0:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.894693 7f797d7fa700 20 rgw meta sync: skipping pending operation 2016-10-25 13:59:21.894697 7f797d7fa700 20 cr:s=0x7f7964767c20:op=0x7f79647e01f0:18RGWMetaSyncShardCR: operate() 2016-10-25 13:59:21.894699 7f797d7fa700 0 rgw meta sync: cr:s=0x7f7964767c20:op=0x7f79647e01f0:18RGWMetaSyncShardCR: adjusting marker pos=1_1477396754.819393_235.1 2016-10-25 13:59:21.894701 7f797d7fa700 20 rgw meta sync: incremental_sync:1594: shard_id=37 mdlog_marker=1_1477396761.789001_270.1 max_marker=1_1477396761.789001_270.1 sync_marker.marker=1_1477396754.819393_235.1 period_marker= 2016-10-25 13:59:21.894705 7f797dffb700 20 link_request req_data=0x7f7964f102d0 req_data->id=531, easy_handle=0x7f7964f104d0 2016-10-25 13:59:21.894710 7f797d7fa700 20 run: stack=0x7f7964767c20 is io blocked 2016-10-25 13:59:21.894712 7f797d7fa700 20 cr:s=0x7f7964ef80e0:op=0x7f7964ef78b0:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.894715 7f797d7fa700 20 run: stack=0x7f7964ef80e0 is done 2016-10-25 13:59:21.895055 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895059 7f797dffb700 10 received header:HTTP/1.1 200 OK 2016-10-25 13:59:21.895060 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895061 7f797dffb700 10 received header:x-amz-request-id: tx0000000000000000004ad-00580f4919-102a-us-east-1 2016-10-25 13:59:21.895065 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895065 7f797dffb700 10 received header:Content-Type: application/json 2016-10-25 13:59:21.895067 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895067 7f797dffb700 10 received header:Content-Length: 1123 2016-10-25 13:59:21.895069 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895070 7f797dffb700 10 received header:Date: Tue, 25 Oct 2016 11:59:21 GMT 2016-10-25 13:59:21.895071 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895071 7f797dffb700 10 received header: 2016-10-25 13:59:21.895091 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ebf130:23RGWReadRemoteMetadataCR: operate() 2016-10-25 13:59:21.895096 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ebe500:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.895102 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ebf130:19RGWMetaStoreEntryCR: operate() 2016-10-25 13:59:21.895105 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ebf130:19RGWMetaStoreEntryCR: operate() 2016-10-25 13:59:21.895110 7f797d7fa700 20 enqueued request req=0x7f7964f0fa10 2016-10-25 13:59:21.895110 7f797d7fa700 20 RGWWQ: 2016-10-25 13:59:21.895111 7f797d7fa700 20 req: 0x7f7964f0fa10 2016-10-25 13:59:21.895113 7f797d7fa700 20 run: stack=0x7f7964ebed30 is io blocked 2016-10-25 13:59:21.895120 7f799d7fa700 20 dequeued request req=0x7f7964f0fa10 2016-10-25 13:59:21.895122 7f799d7fa700 20 RGWWQ: empty 2016-10-25 13:59:21.895339 7f799d7fa700 20 reading from us-west.rgw.data.root:.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 2016-10-25 13:59:21.895347 7f799d7fa700 20 get_system_obj_state: rctx=0x7f799d7f8530 obj=us-west.rgw.data.root:.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 state=0x7f798401cf88 s->prefetch_data=0 2016-10-25 13:59:21.895355 7f799d7fa700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 : hit (requested=22, cached=23) 2016-10-25 13:59:21.895361 7f799d7fa700 20 get_system_obj_state: s->obj_tag was set empty 2016-10-25 13:59:21.895365 7f799d7fa700 20 Read xattr: user.rgw.acl 2016-10-25 13:59:21.895366 7f799d7fa700 20 Read xattr: user.rgw.idtag 2016-10-25 13:59:21.895366 7f799d7fa700 20 Read xattr: user.rgw.manifest 2016-10-25 13:59:21.895369 7f799d7fa700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 : hit (requested=17, cached=23) 2016-10-25 13:59:21.895393 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895397 7f797dffb700 10 received header:HTTP/1.1 404 Not Found 2016-10-25 13:59:21.895398 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895399 7f797dffb700 10 received header:x-amz-request-id: tx0000000000000000004ae-00580f4919-102a-us-east-1 2016-10-25 13:59:21.895402 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895403 7f797dffb700 10 received header:Content-Length: 113 2016-10-25 13:59:21.895404 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895405 7f797dffb700 10 received header:Accept-Ranges: bytes 2016-10-25 13:59:21.895406 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895407 7f797dffb700 10 received header:Content-Type: application/json 2016-10-25 13:59:21.895408 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895409 7f797dffb700 10 received header:Date: Tue, 25 Oct 2016 11:59:21 GMT 2016-10-25 13:59:21.895410 7f797dffb700 10 receive_http_header 2016-10-25 13:59:21.895411 7f797dffb700 10 received header: 2016-10-25 13:59:21.895415 7f799d7fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6676 8.b4be9f65 .meta:bucket.instance:bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26:_SRQ-AXFeaTW9y4BQz7FKH9X:1 [delete,create 0~0,call version.set,writefull 0~341,setxattr user.rgw.acl (127)] snapc 0=[] ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f798400f9d0 con 0x564acad44c40 2016-10-25 13:59:21.895441 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef7310:23RGWReadRemoteMetadataCR: operate() 2016-10-25 13:59:21.895445 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef7310:23RGWReadRemoteMetadataCR: operate() returned r=-2 2016-10-25 13:59:21.895449 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef69e0:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.895454 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef7310:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.895455 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef7310:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.895460 7f797d7fa700 20 enqueued request req=0x7f7964eedc60 2016-10-25 13:59:21.895461 7f797d7fa700 20 RGWWQ: 2016-10-25 13:59:21.895462 7f797d7fa700 20 req: 0x7f7964eedc60 2016-10-25 13:59:21.895464 7f797d7fa700 20 run: stack=0x7f7964eec690 is io blocked 2016-10-25 13:59:21.895469 7f7a01ffb700 20 dequeued request req=0x7f7964eedc60 2016-10-25 13:59:21.895474 7f7a01ffb700 20 RGWWQ: empty 2016-10-25 13:59:21.895488 7f7a01ffb700 20 get_system_obj_state: rctx=0x7f7a01ff9650 obj=us-west.rgw.data.root:bucket26 state=0x7f79e000c098 s->prefetch_data=0 2016-10-25 13:59:21.895495 7f7a01ffb700 10 cache get: name=us-west.rgw.data.root+bucket26 : hit (requested=22, cached=23) 2016-10-25 13:59:21.895500 7f7a01ffb700 20 get_system_obj_state: s->obj_tag was set empty 2016-10-25 13:59:21.895502 7f7a01ffb700 20 Read xattr: user.rgw.idtag 2016-10-25 13:59:21.895503 7f7a01ffb700 20 Read xattr: user.rgw.manifest 2016-10-25 13:59:21.895504 7f7a01ffb700 10 cache get: name=us-west.rgw.data.root+bucket26 : hit (requested=17, cached=23) 2016-10-25 13:59:21.895517 7f7a01ffb700 20 get_system_obj_state: rctx=0x7f7a01ff9650 obj=us-west.rgw.data.root:bucket26 state=0x7f79e000c098 s->prefetch_data=0 2016-10-25 13:59:21.895519 7f7a01ffb700 10 cache get: name=us-west.rgw.data.root+bucket26 : hit (requested=22, cached=23) 2016-10-25 13:59:21.895521 7f7a01ffb700 20 get_system_obj_state: s->obj_tag was set empty 2016-10-25 13:59:21.895522 7f7a01ffb700 10 cache get: name=us-west.rgw.data.root+bucket26 : hit (requested=17, cached=23) 2016-10-25 13:59:21.895539 7f7a01ffb700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6677 7.4a894d1e user1.buckets [call user.remove_bucket] snapc 0=[] ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f79e00065d0 con 0x564acad41190 2016-10-25 13:59:21.910465 7f797d7fa700 20 cr:s=0x7f7964e501a0:op=0x7f7964eb98d0:23RGWReadRemoteMetadataCR: operate() 2016-10-25 13:59:21.910480 7f797d7fa700 20 cr:s=0x7f7964e501a0:op=0x7f7964de58e0:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.910489 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964e7f280:23RGWReadRemoteMetadataCR: operate() 2016-10-25 13:59:21.910491 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964e7f280:23RGWReadRemoteMetadataCR: operate() returned r=-2 2016-10-25 13:59:21.910494 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964ebc960:23RGWReadRemoteMetadataCR: operate() 2016-10-25 13:59:21.910495 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964ebc960:23RGWReadRemoteMetadataCR: operate() returned r=-2 2016-10-25 13:59:21.910498 7f797d7fa700 20 cr:s=0x7f7964e501a0:op=0x7f7964eedf90:19RGWMetaStoreEntryCR: operate() 2016-10-25 13:59:21.910499 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964ebba90:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.910503 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964e89500:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.910506 7f797d7fa700 20 cr:s=0x7f7964e501a0:op=0x7f7964eedf90:19RGWMetaStoreEntryCR: operate() 2016-10-25 13:59:21.910514 7f797d7fa700 20 enqueued request req=0x7f7964eeead0 2016-10-25 13:59:21.910515 7f797d7fa700 20 RGWWQ: 2016-10-25 13:59:21.910516 7f797d7fa700 20 req: 0x7f7964eeead0 2016-10-25 13:59:21.910518 7f797d7fa700 20 run: stack=0x7f7964e501a0 is io blocked 2016-10-25 13:59:21.910520 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964ebc960:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.910523 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964e7f280:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.910524 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964ebc960:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.910527 7f797d7fa700 20 enqueued request req=0x7f7964eef3b0 2016-10-25 13:59:21.910527 7f797d7fa700 20 RGWWQ: 2016-10-25 13:59:21.910527 7f797d7fa700 20 req: 0x7f7964eeead0 2016-10-25 13:59:21.910528 7f797d7fa700 20 req: 0x7f7964eef3b0 2016-10-25 13:59:21.910529 7f797d7fa700 20 run: stack=0x7f7964ebc2c0 is io blocked 2016-10-25 13:59:21.910531 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964e7f280:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.910533 7f797d7fa700 20 enqueued request req=0x7f7964eefbe0 2016-10-25 13:59:21.910533 7f797d7fa700 20 RGWWQ: 2016-10-25 13:59:21.910533 7f797d7fa700 20 req: 0x7f7964eeead0 2016-10-25 13:59:21.910534 7f797d7fa700 20 req: 0x7f7964eef3b0 2016-10-25 13:59:21.910535 7f797d7fa700 20 req: 0x7f7964eefbe0 2016-10-25 13:59:21.910537 7f797d7fa700 20 run: stack=0x7f7964e89d30 is io blocked 2016-10-25 13:59:21.910546 7f79c17fa700 20 dequeued request req=0x7f7964eeead0 2016-10-25 13:59:21.910549 7f79c17fa700 20 RGWWQ: 2016-10-25 13:59:21.910551 7f79c17fa700 20 req: 0x7f7964eef3b0 2016-10-25 13:59:21.910551 7f79c17fa700 20 req: 0x7f7964eefbe0 2016-10-25 13:59:21.910558 7f7a00ff9700 20 dequeued request req=0x7f7964eef3b0 2016-10-25 13:59:21.910561 7f7a00ff9700 20 RGWWQ: 2016-10-25 13:59:21.910562 7f7a00ff9700 20 req: 0x7f7964eefbe0 2016-10-25 13:59:21.910582 7f7a00ff9700 20 get_system_obj_state: rctx=0x7f7a00ff7650 obj=us-west.rgw.data.root:bucket25 state=0x7f79dc00c0f8 s->prefetch_data=0 2016-10-25 13:59:21.910582 7f7a027fc700 20 dequeued request req=0x7f7964eefbe0 2016-10-25 13:59:21.910586 7f7a027fc700 20 RGWWQ: empty 2016-10-25 13:59:21.910590 7f7a00ff9700 10 cache get: name=us-west.rgw.data.root+bucket25 : miss 2016-10-25 13:59:21.910607 7f7a027fc700 20 get_system_obj_state: rctx=0x7f7a027fa650 obj=us-west.rgw.data.root:bucket25 state=0x7f79ec018908 s->prefetch_data=0 2016-10-25 13:59:21.910612 7f7a027fc700 10 cache get: name=us-west.rgw.data.root+bucket25 : miss 2016-10-25 13:59:21.910628 7f7a00ff9700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6678 3.f345cb71 bucket25 [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e29) v7 -- ?+0 0x7f79dc006710 con 0x564acad41190 2016-10-25 13:59:21.910649 7f7a027fc700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6679 3.f345cb71 bucket25 [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e29) v7 -- ?+0 0x7f79ec052fe0 con 0x564acad41190 2016-10-25 13:59:21.910824 7f79c17fa700 20 reading from us-west.rgw.data.root:.bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 2016-10-25 13:59:21.910833 7f79c17fa700 20 get_system_obj_state: rctx=0x7f79c17f8530 obj=us-west.rgw.data.root:.bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 state=0x7f79a401d2d8 s->prefetch_data=0 2016-10-25 13:59:21.910839 7f79c17fa700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 : miss 2016-10-25 13:59:21.910856 7f79c17fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6680 3.f5464dac .bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e29) v7 -- ?+0 0x7f79a400d110 con 0x564acad44c40 2016-10-25 13:59:21.910961 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2953 ==== osd_op_reply(6678 bucket25 [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 212+0+0 (3619899446 0 0) 0x7f7a2000e3c0 con 0x564acad41190 2016-10-25 13:59:21.911021 7f7a00ff9700 10 cache put: name=us-west.rgw.data.root+bucket25 info.flags=0 2016-10-25 13:59:21.911028 7f7a00ff9700 10 adding us-west.rgw.data.root+bucket25 to cache LRU end 2016-10-25 13:59:21.911034 7f7a00ff9700 0 rgw meta sync: ERROR: can't remove key: bucket:bucket25 ret=-2 2016-10-25 13:59:21.911049 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964ebc960:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.911054 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964ebc960:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.911055 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964ebc960:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.911057 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964ebc960:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.911060 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964ebba90:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.911064 7f797d7fa700 20 cr:s=0x7f7964ebc2c0:op=0x7f7964ebba90:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.911067 7f797d7fa700 20 run: stack=0x7f7964ebc2c0 is done 2016-10-25 13:59:21.911092 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2954 ==== osd_op_reply(6679 bucket25 [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 212+0+0 (3619899446 0 0) 0x7f7a20013ba0 con 0x564acad41190 2016-10-25 13:59:21.911122 7f7a027fc700 10 cache put: name=us-west.rgw.data.root+bucket25 info.flags=0 2016-10-25 13:59:21.911128 7f7a027fc700 10 moving us-west.rgw.data.root+bucket25 to cache LRU end 2016-10-25 13:59:21.911134 7f7a027fc700 0 rgw meta sync: ERROR: can't remove key: bucket:bucket25 ret=-2 2016-10-25 13:59:21.911142 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964e7f280:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.911143 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964e7f280:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.911144 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964e7f280:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.911145 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964e7f280:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.911146 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964e89500:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.911148 7f797d7fa700 20 cr:s=0x7f7964e89d30:op=0x7f7964e89500:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.911149 7f797d7fa700 20 run: stack=0x7f7964e89d30 is done 2016-10-25 13:59:21.911218 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3222 ==== osd_op_reply(6680 .bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 [call,getxattrs,stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 270+0+0 (1835481321 0 0) 0x7f7a18015f00 con 0x564acad44c40 2016-10-25 13:59:21.911256 7f79c17fa700 10 cache put: name=us-west.rgw.data.root+.bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 info.flags=0 2016-10-25 13:59:21.911259 7f79c17fa700 10 adding us-west.rgw.data.root+.bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 to cache LRU end 2016-10-25 13:59:21.911294 7f79c17fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6681 8.db17a90e .meta:bucket.instance:bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25:_unvI5l2LCQ44q6k4vsFIsLz:1 [delete,create 0~0,call version.set,writefull 0~341,setxattr user.rgw.acl (127)] snapc 0=[] ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f79a400d110 con 0x564acad41190 2016-10-25 13:59:21.916465 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2955 ==== osd_op_reply(6673 mdlog.sync-status.shard.53 [writefull 0~65] v29'229 uv229 ondisk = 0) v7 ==== 146+0+0 (3006423714 0 0) 0x7f7a20013ba0 con 0x564acad41190 2016-10-25 13:59:21.916495 7f799ffff700 10 cache put: name=us-west.rgw.log+mdlog.sync-status.shard.53 info.flags=1 2016-10-25 13:59:21.916504 7f799ffff700 10 moving us-west.rgw.log+mdlog.sync-status.shard.53 to cache LRU end 2016-10-25 13:59:21.916520 7f799ffff700 10 distributing notification oid=notify.2 bl.length()=307 2016-10-25 13:59:21.916543 7f799ffff700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6682 2.88aa5c95 notify.2 [notify 140160083737328~0] snapc 0=[] ondisk+read+known_if_redirected e29) v7 -- ?+0 0x7f7990021320 con 0x564acad44c40 2016-10-25 13:59:21.916840 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3223 ==== watch-notify(notify (1) cookie 94879230556240 notify 124554052117 ret 0) v3 ==== 349+0+0 (3998292320 0 0) 0x7f7a18025360 con 0x564acad44c40 2016-10-25 13:59:21.916855 7f7a2f7fe700 10 RGWWatcher::handle_notify() notify_id 124554052117 cookie 94879230556240 notifier 4111 bl.length()=307 2016-10-25 13:59:21.916866 7f7a2f7fe700 10 cache put: name=us-west.rgw.log+mdlog.sync-status.shard.53 info.flags=1 2016-10-25 13:59:21.916869 7f7a2f7fe700 10 moving us-west.rgw.log+mdlog.sync-status.shard.53 to cache LRU end 2016-10-25 13:59:21.916870 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3224 ==== osd_op_reply(6682 notify.2 [notify 140160083737328~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+8 (725026202 0 2232870654) 0x7f7a180381b0 con 0x564acad44c40 2016-10-25 13:59:21.916882 7f7a2f7fe700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6683 2.88aa5c95 notify.2 [notify-ack 0~0] snapc 0=[] read+known_if_redirected e29) v7 -- ?+0 0x7f79ec013620 con 0x564acad44c40 2016-10-25 13:59:21.917352 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3225 ==== watch-notify(notify_complete (2) cookie 140160083737328 notify 124554052117 ret 0) v3 ==== 42+0+28 (845197152 0 3773657426) 0x7f7a18025360 con 0x564acad44c40 2016-10-25 13:59:21.917382 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3226 ==== osd_op_reply(6683 notify.2 [notify-ack 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1048150648 0 0) 0x7f7a180381b0 con 0x564acad44c40 2016-10-25 13:59:21.917385 7f797d7fa700 20 cr:s=0x7f7964e7e230:op=0x7f7964e9f280:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.917393 7f797d7fa700 20 cr:s=0x7f7964e7e230:op=0x7f7964e9f280:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.917394 7f797d7fa700 20 cr:s=0x7f7964e7e230:op=0x7f7964e9f280:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.917396 7f797d7fa700 20 cr:s=0x7f7964e7e230:op=0x7f7964e9f280:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.917405 7f797d7fa700 20 cr:s=0x7f7964e7e230:op=0x7f7964de3cb0:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.917409 7f797d7fa700 20 run: stack=0x7f7964e7e230 is done 2016-10-25 13:59:21.930568 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3227 ==== osd_op_reply(6676 .meta:bucket.instance:bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26:_SRQ-AXFeaTW9y4BQz7FKH9X:1 [delete,create 0~0,call,writefull 0~341,setxattr (127)] v29'11 uv11 ondisk = 0) v7 ==== 390+0+0 (3204162658 0 0) 0x7f7a180342e0 con 0x564acad44c40 2016-10-25 13:59:21.930654 7f799d7fa700 10 cache put: name=us-west.rgw.meta+.meta:bucket.instance:bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26:_SRQ-AXFeaTW9y4BQz7FKH9X:1 info.flags=23 2016-10-25 13:59:21.930661 7f799d7fa700 10 moving us-west.rgw.meta+.meta:bucket.instance:bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26:_SRQ-AXFeaTW9y4BQz7FKH9X:1 to cache LRU end 2016-10-25 13:59:21.930665 7f799d7fa700 10 updating xattr: name=user.rgw.acl bl.length()=127 2016-10-25 13:59:21.930667 7f799d7fa700 10 updating xattr: name=user.rgw.idtag bl.length()=0 2016-10-25 13:59:21.930667 7f799d7fa700 10 updating xattr: name=user.rgw.manifest bl.length()=0 2016-10-25 13:59:21.930681 7f799d7fa700 10 distributing notification oid=notify.2 bl.length()=881 2016-10-25 13:59:21.930712 7f799d7fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6684 2.88aa5c95 notify.2 [notify 140159882438672~0] snapc 0=[] ondisk+read+known_if_redirected e29) v7 -- ?+0 0x7f798401e490 con 0x564acad44c40 2016-10-25 13:59:21.930782 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2956 ==== osd_op_reply(6675 mdlog.sync-status.shard.37 [writefull 0~65] v29'234 uv234 ondisk = 0) v7 ==== 146+0+0 (1148521140 0 0) 0x7f7a20013ba0 con 0x564acad41190 2016-10-25 13:59:21.930811 7f79e6ffd700 10 cache put: name=us-west.rgw.log+mdlog.sync-status.shard.37 info.flags=1 2016-10-25 13:59:21.930816 7f79e6ffd700 10 moving us-west.rgw.log+mdlog.sync-status.shard.37 to cache LRU end 2016-10-25 13:59:21.930829 7f79e6ffd700 10 distributing notification oid=notify.2 bl.length()=307 2016-10-25 13:59:21.930854 7f79e6ffd700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6685 2.88aa5c95 notify.2 [notify 140161157491280~0] snapc 0=[] ondisk+read+known_if_redirected e29) v7 -- ?+0 0x7f79d0008b60 con 0x564acad44c40 2016-10-25 13:59:21.930874 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2957 ==== osd_op_reply(6677 user1.buckets [call] v29'84 uv84 ondisk = 0) v7 ==== 133+0+0 (2409521242 0 0) 0x7f7a20013ba0 con 0x564acad41190 2016-10-25 13:59:21.930913 7f7a01ffb700 10 removing us-west.rgw.data.root+bucket26 from cache 2016-10-25 13:59:21.930925 7f7a01ffb700 10 distributing notification oid=notify.0 bl.length()=248 2016-10-25 13:59:21.930941 7f7a01ffb700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6686 2.4322fa9f notify.0 [notify 140161425892816~0] snapc 0=[] ondisk+read+known_if_redirected e29) v7 -- ?+0 0x7f79e000dc10 con 0x564acad44c40 2016-10-25 13:59:21.931134 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3228 ==== watch-notify(notify (1) cookie 94879230556240 notify 124554052118 ret 0) v3 ==== 923+0+0 (424624495 0 0) 0x7f7a18016e90 con 0x564acad44c40 2016-10-25 13:59:21.931170 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3229 ==== osd_op_reply(6684 notify.2 [notify 140159882438672~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+8 (228759458 0 1582457239) 0x7f7a18039520 con 0x564acad44c40 2016-10-25 13:59:21.931220 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3230 ==== watch-notify(notify (1) cookie 94879230552064 notify 124554052119 ret 0) v3 ==== 290+0+0 (3603080168 0 0) 0x7f7a18039520 con 0x564acad44c40 2016-10-25 13:59:21.931233 7f7a2f7fe700 10 RGWWatcher::handle_notify() notify_id 124554052118 cookie 94879230556240 notifier 4111 bl.length()=881 2016-10-25 13:59:21.931249 7f7a2f7fe700 10 cache put: name=us-west.rgw.meta+.meta:bucket.instance:bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26:_SRQ-AXFeaTW9y4BQz7FKH9X:1 info.flags=23 2016-10-25 13:59:21.931253 7f7a2f7fe700 10 moving us-west.rgw.meta+.meta:bucket.instance:bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26:_SRQ-AXFeaTW9y4BQz7FKH9X:1 to cache LRU end 2016-10-25 13:59:21.931256 7f7a2f7fe700 10 updating xattr: name=user.rgw.acl bl.length()=127 2016-10-25 13:59:21.931257 7f7a2f7fe700 10 updating xattr: name=user.rgw.idtag bl.length()=0 2016-10-25 13:59:21.931252 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3231 ==== osd_op_reply(6686 notify.0 [notify 140161425892816~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+8 (4034948727 0 393092272) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.931258 7f7a2f7fe700 10 updating xattr: name=user.rgw.manifest bl.length()=0 2016-10-25 13:59:21.931274 7f7a2f7fe700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6687 2.88aa5c95 notify.2 [notify-ack 0~0] snapc 0=[] read+known_if_redirected e29) v7 -- ?+0 0x7f79ec018870 con 0x564acad44c40 2016-10-25 13:59:21.931285 7f7a2f7fe700 10 RGWWatcher::handle_notify() notify_id 124554052119 cookie 94879230552064 notifier 4111 bl.length()=248 2016-10-25 13:59:21.931297 7f7a2f7fe700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6688 2.4322fa9f notify.0 [notify-ack 0~0] snapc 0=[] read+known_if_redirected e29) v7 -- ?+0 0x7f79ec03f270 con 0x564acad44c40 2016-10-25 13:59:21.931375 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3232 ==== watch-notify(notify (1) cookie 94879230556240 notify 124554052120 ret 0) v3 ==== 349+0+0 (1169952222 0 0) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.931385 7f7a2f7fe700 10 RGWWatcher::handle_notify() notify_id 124554052120 cookie 94879230556240 notifier 4111 bl.length()=307 2016-10-25 13:59:21.931392 7f7a2f7fe700 10 cache put: name=us-west.rgw.log+mdlog.sync-status.shard.37 info.flags=1 2016-10-25 13:59:21.931394 7f7a2f7fe700 10 moving us-west.rgw.log+mdlog.sync-status.shard.37 to cache LRU end 2016-10-25 13:59:21.931404 7f7a2f7fe700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6689 2.88aa5c95 notify.2 [notify-ack 0~0] snapc 0=[] read+known_if_redirected e29) v7 -- ?+0 0x7f79ec03f270 con 0x564acad44c40 2016-10-25 13:59:21.931434 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3233 ==== osd_op_reply(6685 notify.2 [notify 140161157491280~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+8 (1854215345 0 2936806766) 0x7f7a18016e90 con 0x564acad44c40 2016-10-25 13:59:21.931596 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3234 ==== watch-notify(notify_complete (2) cookie 140159882438672 notify 124554052118 ret 0) v3 ==== 42+0+28 (2967796570 0 3773657426) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.931630 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3235 ==== osd_op_reply(6687 notify.2 [notify-ack 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1048150648 0 0) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.931663 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3236 ==== watch-notify(notify_complete (2) cookie 140161425892816 notify 124554052119 ret 0) v3 ==== 42+0+28 (431385752 0 3402964263) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.931663 7f799d7fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6690 3.8c4e20d2 .bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 [delete,create 0~0,call version.check_conds,call version.set,writefull 0~341,setxattr user.rgw.acl (127)] snapc 0=[] ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f798401bc60 con 0x564acad44c40 2016-10-25 13:59:21.931684 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3237 ==== osd_op_reply(6688 notify.0 [notify-ack 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1984897511 0 0) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.931712 7f7a01ffb700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6691 3.c8ef106b bucket26 [call version.check_conds,call version.set,delete] snapc 0=[] ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f79e0018850 con 0x564acad41190 2016-10-25 13:59:21.932034 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3238 ==== watch-notify(notify_complete (2) cookie 140161157491280 notify 124554052120 ret 0) v3 ==== 42+0+28 (1697573312 0 3773657426) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.932069 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3239 ==== osd_op_reply(6689 notify.2 [notify-ack 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1048150648 0 0) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.932080 7f797d7fa700 20 cr:s=0x7f7964e4d5d0:op=0x7f7964e4d9b0:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.932085 7f797d7fa700 20 cr:s=0x7f7964e4d5d0:op=0x7f7964e4d9b0:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.932087 7f797d7fa700 20 cr:s=0x7f7964e4d5d0:op=0x7f7964e4d9b0:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.932088 7f797d7fa700 20 cr:s=0x7f7964e4d5d0:op=0x7f7964e4d9b0:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.932097 7f797d7fa700 20 cr:s=0x7f7964e4d5d0:op=0x7f7964e99670:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.932099 7f797d7fa700 20 run: stack=0x7f7964e4d5d0 is done 2016-10-25 13:59:21.936725 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2958 ==== osd_op_reply(6681 .meta:bucket.instance:bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25:_unvI5l2LCQ44q6k4vsFIsLz:1 [delete,create 0~0,call,writefull 0~341,setxattr (127)] v29'13 uv13 ondisk = 0) v7 ==== 390+0+0 (1183504357 0 0) 0x7f7a20020d40 con 0x564acad41190 2016-10-25 13:59:21.936757 7f79c17fa700 10 cache put: name=us-west.rgw.meta+.meta:bucket.instance:bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25:_unvI5l2LCQ44q6k4vsFIsLz:1 info.flags=23 2016-10-25 13:59:21.936763 7f79c17fa700 10 moving us-west.rgw.meta+.meta:bucket.instance:bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25:_unvI5l2LCQ44q6k4vsFIsLz:1 to cache LRU end 2016-10-25 13:59:21.936767 7f79c17fa700 10 updating xattr: name=user.rgw.acl bl.length()=127 2016-10-25 13:59:21.936768 7f79c17fa700 10 updating xattr: name=user.rgw.idtag bl.length()=0 2016-10-25 13:59:21.936769 7f79c17fa700 10 updating xattr: name=user.rgw.manifest bl.length()=0 2016-10-25 13:59:21.936782 7f79c17fa700 10 distributing notification oid=notify.6 bl.length()=881 2016-10-25 13:59:21.936800 7f79c17fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6692 2.97c520d4 notify.6 [notify 140160419277888~0] snapc 0=[] ondisk+read+known_if_redirected e29) v7 -- ?+0 0x7f79a4016990 con 0x564acad44c40 2016-10-25 13:59:21.937104 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3240 ==== watch-notify(notify (1) cookie 94879230566576 notify 124554052121 ret 0) v3 ==== 923+0+0 (2452462787 0 0) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.937121 7f7a2f7fe700 10 RGWWatcher::handle_notify() notify_id 124554052121 cookie 94879230566576 notifier 4111 bl.length()=881 2016-10-25 13:59:21.937136 7f7a2f7fe700 10 cache put: name=us-west.rgw.meta+.meta:bucket.instance:bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25:_unvI5l2LCQ44q6k4vsFIsLz:1 info.flags=23 2016-10-25 13:59:21.937140 7f7a2f7fe700 10 moving us-west.rgw.meta+.meta:bucket.instance:bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25:_unvI5l2LCQ44q6k4vsFIsLz:1 to cache LRU end 2016-10-25 13:59:21.937143 7f7a2f7fe700 10 updating xattr: name=user.rgw.acl bl.length()=127 2016-10-25 13:59:21.937144 7f7a2f7fe700 10 updating xattr: name=user.rgw.idtag bl.length()=0 2016-10-25 13:59:21.937145 7f7a2f7fe700 10 updating xattr: name=user.rgw.manifest bl.length()=0 2016-10-25 13:59:21.937143 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3241 ==== osd_op_reply(6692 notify.6 [notify 140160419277888~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+8 (337561402 0 3861930057) 0x7f7a18035540 con 0x564acad44c40 2016-10-25 13:59:21.937159 7f7a2f7fe700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6693 2.97c520d4 notify.6 [notify-ack 0~0] snapc 0=[] read+known_if_redirected e29) v7 -- ?+0 0x7f79ec03f270 con 0x564acad44c40 2016-10-25 13:59:21.937583 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3242 ==== watch-notify(notify_complete (2) cookie 140160419277888 notify 124554052121 ret 0) v3 ==== 42+0+28 (1473625675 0 3459847823) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.937612 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3243 ==== osd_op_reply(6693 notify.6 [notify-ack 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1562312443 0 0) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.937626 7f79c17fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6694 3.f5464dac .bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 [delete,create 0~0,call version.set,writefull 0~341,setxattr user.rgw.acl (127)] snapc 0=[] ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f79a400ac40 con 0x564acad44c40 2016-10-25 13:59:21.957516 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2959 ==== osd_op_reply(6691 bucket26 [call,call,delete] v29'16 uv16 ondisk = 0) v7 ==== 212+0+0 (35943619 0 0) 0x7f7a20013ba0 con 0x564acad41190 2016-10-25 13:59:21.957590 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef7310:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.957599 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef7310:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.957601 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef7310:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.957602 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef7310:20RGWMetaRemoveEntryCR: operate() 2016-10-25 13:59:21.957603 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef69e0:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.957609 7f797d7fa700 20 cr:s=0x7f7964eec690:op=0x7f7964ef69e0:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.957614 7f797d7fa700 20 run: stack=0x7f7964eec690 is done 2016-10-25 13:59:21.972370 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3244 ==== osd_op_reply(6690 .bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 [delete,create 0~0,call,call,writefull 0~341,setxattr (127)] v29'11 uv11 ondisk = 0) v7 ==== 396+0+0 (2097133766 0 0) 0x7f7a18024050 con 0x564acad44c40 2016-10-25 13:59:21.972417 7f799d7fa700 10 cache put: name=us-west.rgw.data.root+.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 info.flags=23 2016-10-25 13:59:21.972432 7f799d7fa700 10 moving us-west.rgw.data.root+.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 to cache LRU end 2016-10-25 13:59:21.972439 7f799d7fa700 10 updating xattr: name=user.rgw.acl bl.length()=127 2016-10-25 13:59:21.972441 7f799d7fa700 10 updating xattr: name=user.rgw.idtag bl.length()=0 2016-10-25 13:59:21.972443 7f799d7fa700 10 updating xattr: name=user.rgw.manifest bl.length()=0 2016-10-25 13:59:21.972460 7f799d7fa700 10 distributing notification oid=notify.6 bl.length()=865 2016-10-25 13:59:21.972498 7f799d7fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6695 2.97c520d4 notify.6 [notify 140159882479056~0] snapc 0=[] ondisk+read+known_if_redirected e29) v7 -- ?+0 0x7f798401e6d0 con 0x564acad44c40 2016-10-25 13:59:21.972537 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3245 ==== osd_op_reply(6694 .bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 [delete,create 0~0,call,writefull 0~341,setxattr (127)] v29'26 uv26 ondisk = 0) v7 ==== 354+0+0 (995517979 0 0) 0x7f7a18024050 con 0x564acad44c40 2016-10-25 13:59:21.972565 7f79c17fa700 10 cache put: name=us-west.rgw.data.root+.bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 info.flags=23 2016-10-25 13:59:21.972570 7f79c17fa700 10 moving us-west.rgw.data.root+.bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 to cache LRU end 2016-10-25 13:59:21.972573 7f79c17fa700 10 updating xattr: name=user.rgw.acl bl.length()=127 2016-10-25 13:59:21.972573 7f79c17fa700 10 updating xattr: name=user.rgw.idtag bl.length()=0 2016-10-25 13:59:21.972574 7f79c17fa700 10 updating xattr: name=user.rgw.manifest bl.length()=0 2016-10-25 13:59:21.972583 7f79c17fa700 10 distributing notification oid=notify.6 bl.length()=865 2016-10-25 13:59:21.972599 7f79c17fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6696 2.97c520d4 notify.6 [notify 140160419277888~0] snapc 0=[] ondisk+read+known_if_redirected e29) v7 -- ?+0 0x7f79a401d7d0 con 0x564acad44c40 2016-10-25 13:59:21.972826 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3246 ==== watch-notify(notify (1) cookie 94879230566576 notify 124554052122 ret 0) v3 ==== 907+0+0 (994017759 0 0) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.972843 7f7a2f7fe700 10 RGWWatcher::handle_notify() notify_id 124554052122 cookie 94879230566576 notifier 4111 bl.length()=865 2016-10-25 13:59:21.972857 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3247 ==== osd_op_reply(6695 notify.6 [notify 140159882479056~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+8 (2652084434 0 1031071520) 0x7f7a18035f70 con 0x564acad44c40 2016-10-25 13:59:21.972861 7f7a2f7fe700 10 cache put: name=us-west.rgw.data.root+.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 info.flags=23 2016-10-25 13:59:21.972864 7f7a2f7fe700 10 moving us-west.rgw.data.root+.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 to cache LRU end 2016-10-25 13:59:21.972866 7f7a2f7fe700 10 updating xattr: name=user.rgw.acl bl.length()=127 2016-10-25 13:59:21.972867 7f7a2f7fe700 10 updating xattr: name=user.rgw.idtag bl.length()=0 2016-10-25 13:59:21.972868 7f7a2f7fe700 10 updating xattr: name=user.rgw.manifest bl.length()=0 2016-10-25 13:59:21.972883 7f7a2f7fe700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6697 2.97c520d4 notify.6 [notify-ack 0~0] snapc 0=[] read+known_if_redirected e29) v7 -- ?+0 0x7f79ec040f50 con 0x564acad44c40 2016-10-25 13:59:21.972979 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3248 ==== watch-notify(notify (1) cookie 94879230566576 notify 124554052123 ret 0) v3 ==== 907+0+0 (2342341402 0 0) 0x7f7a18034ad0 con 0x564acad44c40 2016-10-25 13:59:21.972992 7f7a2f7fe700 10 RGWWatcher::handle_notify() notify_id 124554052123 cookie 94879230566576 notifier 4111 bl.length()=865 2016-10-25 13:59:21.973003 7f7a2f7fe700 10 cache put: name=us-west.rgw.data.root+.bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 info.flags=23 2016-10-25 13:59:21.973006 7f7a2f7fe700 10 moving us-west.rgw.data.root+.bucket.meta.bucket25:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 to cache LRU end 2016-10-25 13:59:21.973008 7f7a2f7fe700 10 updating xattr: name=user.rgw.acl bl.length()=127 2016-10-25 13:59:21.973009 7f7a2f7fe700 10 updating xattr: name=user.rgw.idtag bl.length()=0 2016-10-25 13:59:21.973009 7f7a2f7fe700 10 updating xattr: name=user.rgw.manifest bl.length()=0 2016-10-25 13:59:21.973006 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3249 ==== osd_op_reply(6696 notify.6 [notify 140160419277888~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+8 (337561402 0 1950915079) 0x7f7a18042c20 con 0x564acad44c40 2016-10-25 13:59:21.973022 7f7a2f7fe700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6698 2.97c520d4 notify.6 [notify-ack 0~0] snapc 0=[] read+known_if_redirected e29) v7 -- ?+0 0x7f79ec04d050 con 0x564acad44c40 2016-10-25 13:59:21.973145 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3250 ==== watch-notify(notify_complete (2) cookie 140159882479056 notify 124554052122 ret 0) v3 ==== 42+0+28 (2430184390 0 3459847823) 0x7f7a18042c20 con 0x564acad44c40 2016-10-25 13:59:21.973172 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3251 ==== osd_op_reply(6697 notify.6 [notify-ack 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1562312443 0 0) 0x7f7a18042c20 con 0x564acad44c40 2016-10-25 13:59:21.973206 7f799d7fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6699 9.20175489 .dir.0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 [create 0~0 [excl],call rgw.bucket_init_index] snapc 0=[] ack+ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f7984016ff0 con 0x564acad41190 2016-10-25 13:59:21.973508 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2960 ==== osd_op_reply(6699 .dir.0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 [create 0~0 [excl],call] v0'0 uv0 ondisk = -17 ((17) File exists)) v7 ==== 211+0+0 (3751070808 0 0) 0x7f7a20013ba0 con 0x564acad41190 2016-10-25 13:59:21.973551 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3252 ==== watch-notify(notify_complete (2) cookie 140160419277888 notify 124554052123 ret 0) v3 ==== 42+0+28 (648433062 0 3459847823) 0x7f7a18042c20 con 0x564acad44c40 2016-10-25 13:59:21.973578 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3253 ==== osd_op_reply(6698 notify.6 [notify-ack 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1562312443 0 0) 0x7f7a18042c20 con 0x564acad44c40 2016-10-25 13:59:21.973858 7f79c17fa700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6700 9.235ab709 .dir.0474fe09-15db-4e91-8f51-ff360bdeb074.4138.25 [create 0~0 [excl],call rgw.bucket_init_index] snapc 0=[] ack+ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f79a400b280 con 0x564acad41190 2016-10-25 13:59:21.973894 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ebf130:19RGWMetaStoreEntryCR: operate() 2016-10-25 13:59:21.973900 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ebf130:19RGWMetaStoreEntryCR: operate() 2016-10-25 13:59:21.973901 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ebf130:19RGWMetaStoreEntryCR: operate() 2016-10-25 13:59:21.973903 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ebf130:19RGWMetaStoreEntryCR: operate() 2016-10-25 13:59:21.973904 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ebe500:24RGWMetaSyncSingleEntryCR: operate() 2016-10-25 13:59:21.973909 7f797d7fa700 20 rgw meta sync: store_marker(): updating marker marker_oid=mdlog.sync-status.shard.37 marker=1_1477396761.789001_270.1 2016-10-25 13:59:21.973926 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ef69e0:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.973929 7f797d7fa700 20 cr:s=0x7f7964ebed30:op=0x7f7964ef69e0:21RGWSimpleRadosWriteCRI20rgw_meta_sync_markerE: operate() 2016-10-25 13:59:21.973938 7f797d7fa700 20 enqueued request req=0x7f7964e89ba0 2016-10-25 13:59:21.973940 7f797d7fa700 20 RGWWQ: 2016-10-25 13:59:21.973940 7f797d7fa700 20 req: 0x7f7964e89ba0 2016-10-25 13:59:21.973944 7f797d7fa700 20 run: stack=0x7f7964ebed30 is io blocked 2016-10-25 13:59:21.973946 7f799f7fe700 20 dequeued request req=0x7f7964e89ba0 2016-10-25 13:59:21.973950 7f799f7fe700 20 RGWWQ: empty 2016-10-25 13:59:21.973970 7f799f7fe700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6701 5.2d1ff4fd mdlog.sync-status.shard.37 [writefull 0~65] snapc 0=[] ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f7994031980 con 0x564acad41190 2016-10-25 13:59:21.997236 7f7920ff1700 10 receive_http_header 2016-10-25 13:59:21.997245 7f7920ff1700 10 received header:HTTP/1.1 204 No Content 2016-10-25 13:59:21.997248 7f7920ff1700 10 receive_http_header 2016-10-25 13:59:21.997249 7f7920ff1700 10 received header:x-amz-request-id: tx0000000000000000004a6-00580f4919-102a-us-east-1 2016-10-25 13:59:21.997254 7f7920ff1700 10 receive_http_header 2016-10-25 13:59:21.997255 7f7920ff1700 10 received header:Date: Tue, 25 Oct 2016 11:59:21 GMT 2016-10-25 13:59:21.997257 7f7920ff1700 10 receive_http_header 2016-10-25 13:59:21.997257 7f7920ff1700 10 received header: 2016-10-25 13:59:21.997338 7f7920ff1700 20 response: 2016-10-25 13:59:21.997356 7f7920ff1700 20 reading from us-west.rgw.data.root:.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 2016-10-25 13:59:21.997368 7f7920ff1700 20 get_system_obj_state: rctx=0x7f7920fed9a0 obj=us-west.rgw.data.root:.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 state=0x7f7970030208 s->prefetch_data=0 2016-10-25 13:59:21.997376 7f7920ff1700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 : hit (requested=22, cached=23) 2016-10-25 13:59:21.997384 7f7920ff1700 20 get_system_obj_state: s->obj_tag was set empty 2016-10-25 13:59:21.997386 7f7920ff1700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 : hit (requested=17, cached=23) 2016-10-25 13:59:21.997399 7f7920ff1700 10 cls_bucket_list bucket26(@{i=us-west.rgw.buckets.index,e=us-west.rgw.buckets.non-ec}us-west.rgw.buckets.data[0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26]) start [] num_entries 1000 2016-10-25 13:59:21.997404 7f7920ff1700 20 reading from us-west.rgw.data.root:.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 2016-10-25 13:59:21.997407 7f7920ff1700 20 get_system_obj_state: rctx=0x7f7920fed3e0 obj=us-west.rgw.data.root:.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 state=0x7f797002e688 s->prefetch_data=0 2016-10-25 13:59:21.997410 7f7920ff1700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 : hit (requested=22, cached=23) 2016-10-25 13:59:21.997412 7f7920ff1700 20 get_system_obj_state: s->obj_tag was set empty 2016-10-25 13:59:21.997413 7f7920ff1700 10 cache get: name=us-west.rgw.data.root+.bucket.meta.bucket26:0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 : hit (requested=17, cached=23) 2016-10-25 13:59:21.997445 7f7920ff1700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6702 9.20175489 .dir.0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 [call rgw.bucket_list] snapc 0=[] ack+read+known_if_redirected e29) v7 -- ?+0 0x7f797002f200 con 0x564acad41190 2016-10-25 13:59:21.997937 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2961 ==== osd_op_reply(6702 .dir.0474fe09-15db-4e91-8f51-ff360bdeb074.4138.26 [call] v0'0 uv11 ondisk = 0) v7 ==== 169+0+102 (2507748447 0 1024305959) 0x7f7a20013ba0 con 0x564acad41190 2016-10-25 13:59:21.997987 7f7920ff1700 10 distributing notification oid=notify.0 bl.length()=248 2016-10-25 13:59:21.998005 7f7920ff1700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6703 2.4322fa9f notify.0 [notify 140159547008976~0] snapc 0=[] ondisk+read+known_if_redirected e29) v7 -- ?+0 0x7f797002f200 con 0x564acad44c40 2016-10-25 13:59:21.998302 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3254 ==== watch-notify(notify (1) cookie 94879230552064 notify 124554052124 ret 0) v3 ==== 290+0+0 (1154329996 0 0) 0x7f7a18042c20 con 0x564acad44c40 2016-10-25 13:59:21.998321 7f7a2f7fe700 10 RGWWatcher::handle_notify() notify_id 124554052124 cookie 94879230552064 notifier 4111 bl.length()=248 2016-10-25 13:59:21.998334 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3255 ==== osd_op_reply(6703 notify.0 [notify 140159547008976~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+8 (2705112419 0 2383519491) 0x7f7a18043810 con 0x564acad44c40 2016-10-25 13:59:21.998351 7f7a2f7fe700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6816/27634 -- osd_op(client.4111.0:6704 2.4322fa9f notify.0 [notify-ack 0~0] snapc 0=[] read+known_if_redirected e29) v7 -- ?+0 0x7f79ec04be00 con 0x564acad44c40 2016-10-25 13:59:21.998847 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3256 ==== watch-notify(notify_complete (2) cookie 140159547008976 notify 124554052124 ret 0) v3 ==== 42+0+28 (752535219 0 3402964263) 0x7f7a18043810 con 0x564acad44c40 2016-10-25 13:59:21.998879 7f7a48215700 1 -- 127.0.0.1:0/1778501402 <== osd.1 127.0.0.1:6816/27634 3257 ==== osd_op_reply(6704 notify.0 [notify-ack 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (1984897511 0 0) 0x7f7a18043810 con 0x564acad44c40 2016-10-25 13:59:21.998892 7f7920ff1700 1 -- 127.0.0.1:0/1778501402 --> 127.0.0.1:6812/27412 -- osd_op(client.4111.0:6705 3.c8ef106b bucket26 [call version.check_conds,call version.set,delete] snapc 0=[] ondisk+write+known_if_redirected e29) v7 -- ?+0 0x7f7970039760 con 0x564acad41190 2016-10-25 13:59:21.999214 7f7a48417700 1 -- 127.0.0.1:0/1778501402 <== osd.0 127.0.0.1:6812/27412 2962 ==== osd_op_reply(6705 bucket26 [call,call,delete] v0'0 uv0 ondisk = -125 ((125) Operation canceled)) v7 ==== 212+0+0 (1575458175 0 0) 0x7f7a20013ba0 con 0x564acad41190 2016-10-25 13:59:21.999244 7f7920ff1700 2 req 1192:0.312583:s3:DELETE /bucket26/:delete_bucket:completing 2016-10-25 13:59:21.999247 7f7920ff1700 0 WARNING: set_req_state_err err_no=125 resorting to 500 2016-10-25 13:59:21.999286 7f7920ff1700 2 req 1192:0.312625:s3:DELETE /bucket26/:delete_bucket:op status=-125 2016-10-25 13:59:21.999289 7f7920ff1700 2 req 1192:0.312628:s3:DELETE /bucket26/:delete_bucket:http status=500 2016-10-25 13:59:21.999291 7f7920ff1700 1 ====== req done req=0x7f7920fee7e0 op status=-125 http_status=500 ======