Project

General

Profile

Bug #21619 » luminous-rgw-iccluster015.log

Radosgw debug log #1 - Yoann Moulin, 10/02/2017 01:32 PM

 
2017-10-02 06:25:01.251585 7f4544a83700 1 rgw realm reloader: Pausing frontends for realm update...
2017-10-02 06:25:01.251606 7f4544a83700 1 rgw realm reloader: Frontends paused
2017-10-02 06:25:01.251949 7f4554caa700 0 ERROR: failed to clone shard, completion_mgr.get_next() returned ret=-125
2017-10-02 06:25:01.303535 7f4544a83700 1 rgw realm reloader: Store closed
2017-10-02 06:25:02.080988 7f4544a83700 1 rgw realm reloader: Creating new store
2017-10-02 06:25:02.081255 7f4544a83700 1 mgrc service_daemon_register rgw.iccluster015 metadata {arch=x86_64,ceph_version=ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable),cpu=Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz,distro=ubuntu,distro_description=Ubuntu 16.04.3 LTS,distro_version=16.04,frontend_config#0=civetweb port=127.0.0.1:8080 num_threads=24,frontend_type#0=civetweb,hostname=iccluster015,kernel_description=#116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017,kernel_version=4.4.0-93-generic,mem_swap_kb=2000056,mem_total_kb=131920952,num_handles=1,os=Linux,pid=2593,zone_id=69d2fd65-fcf9-461b-865f-3dbb053803c4,zone_name=default,zonegroup_id=43d23097-56b9-48a6-ad52-de42341be4bd,zonegroup_name=default}
2017-10-02 06:25:02.081317 7f4544a83700 1 rgw realm reloader: Finishing initialization of new store
2017-10-02 06:25:02.081320 7f4544a83700 1 rgw realm reloader: - REST subsystem init
2017-10-02 06:25:02.081356 7f4544a83700 1 rgw realm reloader: - user subsystem init
2017-10-02 06:25:02.081365 7f4544a83700 1 rgw realm reloader: - user subsystem init
2017-10-02 06:25:02.081371 7f4544a83700 1 rgw realm reloader: - usage subsystem init
2017-10-02 06:25:02.081442 7f4544a83700 1 rgw realm reloader: Resuming frontends with new realm configuration.
2017-10-02 06:34:38.019975 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 06:49:38.020098 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 07:04:38.020193 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 07:19:38.020324 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 07:34:38.020452 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 07:49:38.020588 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 08:04:38.020725 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 08:19:38.020853 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 08:34:38.020979 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 08:49:38.021105 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 09:04:38.021249 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 09:19:38.021402 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 09:34:38.021529 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 09:49:38.021655 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 10:04:38.021794 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 10:19:38.021936 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 10:34:38.022065 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 10:49:38.022190 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 11:04:38.022332 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 11:19:38.022481 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 11:34:38.022644 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 11:49:38.022767 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 12:04:38.022853 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 12:19:38.022992 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 12:34:38.023120 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 12:49:38.023246 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 13:04:38.023393 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 13:19:38.023540 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 13:34:38.023669 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 13:49:38.023811 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 14:01:11.273784 7f4543280700 0 WARNING: RGWReshard::add failed to drop lock on image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.44353.1 ret=-2
2017-10-02 14:04:38.023974 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 14:19:38.024102 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 14:34:38.024224 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 14:49:38.024372 7f45524a5700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 14:52:04.685732 7f4551a9d700 -1 received signal: Terminated from PID: 1 task name: /sbin/init splash UID: 0
2017-10-02 14:52:04.685760 7f4551a9d700 1 handle_sigterm
2017-10-02 14:52:04.685785 7f45830bbe80 -1 shutting down
2017-10-02 14:52:04.685805 7f4551a9d700 1 handle_sigterm set alarm for 120
2017-10-02 14:52:04.807443 7f4554caa700 0 ERROR: failed to clone shard, completion_mgr.get_next() returned ret=-125
2017-10-02 14:52:04.859550 7f45830bbe80 1 final shutdown
2017-10-02 14:52:05.006347 7f9d36440e80 0 deferred set uid:gid to 64045:64045 (ceph:ceph)
2017-10-02 14:52:05.006388 7f9d36440e80 0 ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable), process (unknown), pid 41860
2017-10-02 14:52:05.014421 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b8840 obj=.rgw.root:default.realm state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.014432 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:52:05.016728 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.016743 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.017308 7f9d36440e80 20 rados->read r=0 bl.length=46
2017-10-02 14:52:05.017362 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b88c0 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.018736 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.018753 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.019464 7f9d36440e80 20 rados->read r=0 bl.length=107
2017-10-02 14:52:05.019520 7f9d36440e80 20 realm default b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
2017-10-02 14:52:05.019544 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b86b0 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.020329 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.020347 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.021044 7f9d36440e80 20 rados->read r=0 bl.length=107
2017-10-02 14:52:05.021083 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b8820 obj=.rgw.root:periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.022405 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.022421 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.023035 7f9d36440e80 20 rados->read r=0 bl.length=10
2017-10-02 14:52:05.023087 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b8840 obj=.rgw.root:periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.023893 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.023926 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.024615 7f9d36440e80 20 rados->read r=0 bl.length=846
2017-10-02 14:52:05.024686 7f9d36440e80 20 current period e7bfcb5a-829b-418f-ae26-d6573a5cc8b9
2017-10-02 14:52:05.024706 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b7f90 obj=.rgw.root:converted state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.026407 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b73e0 obj=.rgw.root:default.realm state=0x526954ee60 s->prefetch_data=0
2017-10-02 14:52:05.027291 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.027309 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.028023 7f9d36440e80 20 rados->read r=0 bl.length=46
2017-10-02 14:52:05.028055 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b7460 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x526954ee60 s->prefetch_data=0
2017-10-02 14:52:05.028836 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.028854 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.029581 7f9d36440e80 20 rados->read r=0 bl.length=107
2017-10-02 14:52:05.029625 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b7570 obj=.rgw.root:default.region state=0x526954ee60 s->prefetch_data=0
2017-10-02 14:52:05.286714 7f9d36440e80 20 RGWRados::pool_iterate: got zonegroup_info.43d23097-56b9-48a6-ad52-de42341be4bd
2017-10-02 14:52:05.286734 7f9d36440e80 20 RGWRados::pool_iterate: got periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch
2017-10-02 14:52:05.286738 7f9d36440e80 20 RGWRados::pool_iterate: got periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1
2017-10-02 14:52:05.286739 7f9d36440e80 20 RGWRados::pool_iterate: got realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
2017-10-02 14:52:05.286742 7f9d36440e80 20 RGWRados::pool_iterate: got default.realm
2017-10-02 14:52:05.286744 7f9d36440e80 20 RGWRados::pool_iterate: got default.zone.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
2017-10-02 14:52:05.286745 7f9d36440e80 20 RGWRados::pool_iterate: got periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch
2017-10-02 14:52:05.286747 7f9d36440e80 20 RGWRados::pool_iterate: got default.zonegroup.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
2017-10-02 14:52:05.286751 7f9d36440e80 20 RGWRados::pool_iterate: got periods.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a:staging
2017-10-02 14:52:05.286753 7f9d36440e80 20 RGWRados::pool_iterate: got realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a.control
2017-10-02 14:52:05.286754 7f9d36440e80 20 RGWRados::pool_iterate: got period_config.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
2017-10-02 14:52:05.286756 7f9d36440e80 20 RGWRados::pool_iterate: got zone_info.69d2fd65-fcf9-461b-865f-3dbb053803c4
2017-10-02 14:52:05.286758 7f9d36440e80 20 RGWRados::pool_iterate: got realms_names.default
2017-10-02 14:52:05.286759 7f9d36440e80 20 RGWRados::pool_iterate: got zone_names.default
2017-10-02 14:52:05.286780 7f9d36440e80 20 RGWRados::pool_iterate: got zonegroups_names.default
2017-10-02 14:52:05.286782 7f9d36440e80 20 RGWRados::pool_iterate: got periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1
2017-10-02 14:52:05.286803 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b7660 obj=.rgw.root:zone_names.default state=0x526954ee60 s->prefetch_data=0
2017-10-02 14:52:05.287772 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.287791 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.305521 7f9d36440e80 20 rados->read r=0 bl.length=46
2017-10-02 14:52:05.305592 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b7660 obj=.rgw.root:zone_info.69d2fd65-fcf9-461b-865f-3dbb053803c4 state=0x526954ee60 s->prefetch_data=0
2017-10-02 14:52:05.306445 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.306462 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.307154 7f9d36440e80 20 rados->read r=0 bl.length=724
2017-10-02 14:52:05.307203 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b7660 obj=.rgw.root:zonegroups_names.default state=0x526954ee60 s->prefetch_data=0
2017-10-02 14:52:05.308231 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.308249 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.309105 7f9d36440e80 20 rados->read r=0 bl.length=46
2017-10-02 14:52:05.309156 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b7660 obj=.rgw.root:zonegroup_info.43d23097-56b9-48a6-ad52-de42341be4bd state=0x526954ee60 s->prefetch_data=0
2017-10-02 14:52:05.310189 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.310207 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.311113 7f9d36440e80 20 rados->read r=0 bl.length=369
2017-10-02 14:52:05.311174 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b8a50 obj=.rgw.root:region_map state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.312167 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b80d0 obj=.rgw.root:default.realm state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.313148 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.313166 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.313991 7f9d36440e80 20 rados->read r=0 bl.length=46
2017-10-02 14:52:05.314027 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b8150 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.314849 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.314867 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.315585 7f9d36440e80 20 rados->read r=0 bl.length=107
2017-10-02 14:52:05.315642 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b8260 obj=.rgw.root:default.zonegroup.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.316487 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.316506 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.317291 7f9d36440e80 20 rados->read r=0 bl.length=46
2017-10-02 14:52:05.317339 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b84e0 obj=.rgw.root:zonegroup_info.43d23097-56b9-48a6-ad52-de42341be4bd state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.318232 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.318250 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.319103 7f9d36440e80 20 rados->read r=0 bl.length=369
2017-10-02 14:52:05.319147 7f9d36440e80 20 period zonegroup init ret 0
2017-10-02 14:52:05.319152 7f9d36440e80 20 period zonegroup name default
2017-10-02 14:52:05.319155 7f9d36440e80 20 using current period zonegroup default
2017-10-02 14:52:05.319167 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b80d0 obj=.rgw.root:default.realm state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.320075 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.320092 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.320907 7f9d36440e80 20 rados->read r=0 bl.length=46
2017-10-02 14:52:05.320945 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b8150 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.321741 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.321759 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.322467 7f9d36440e80 20 rados->read r=0 bl.length=107
2017-10-02 14:52:05.322507 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b8260 obj=.rgw.root:default.zone.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.323361 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.323379 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.324207 7f9d36440e80 20 rados->read r=0 bl.length=46
2017-10-02 14:52:05.324247 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b84e0 obj=.rgw.root:zone_info.69d2fd65-fcf9-461b-865f-3dbb053803c4 state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.325101 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.325131 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.325844 7f9d36440e80 20 rados->read r=0 bl.length=724
2017-10-02 14:52:05.325902 7f9d36440e80 10 Cannot find current period zone using local zone
2017-10-02 14:52:05.325915 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b88c0 obj=.rgw.root:zone_info.69d2fd65-fcf9-461b-865f-3dbb053803c4 state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.326664 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.326682 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.327399 7f9d36440e80 20 rados->read r=0 bl.length=724
2017-10-02 14:52:05.327445 7f9d36440e80 20 zone default
2017-10-02 14:52:05.335398 7f9d36440e80 20 add_watcher() i=0
2017-10-02 14:52:05.342494 7f9d36440e80 20 add_watcher() i=1
2017-10-02 14:52:05.350839 7f9d36440e80 20 add_watcher() i=2
2017-10-02 14:52:05.359321 7f9d36440e80 20 add_watcher() i=3
2017-10-02 14:52:05.366734 7f9d36440e80 20 add_watcher() i=4
2017-10-02 14:52:05.374339 7f9d36440e80 20 add_watcher() i=5
2017-10-02 14:52:05.382919 7f9d36440e80 20 add_watcher() i=6
2017-10-02 14:52:05.390625 7f9d36440e80 20 add_watcher() i=7
2017-10-02 14:52:05.390641 7f9d36440e80 2 all 8 watchers are set, enabling cache
2017-10-02 14:52:05.390811 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b81d0 obj=default.rgw.log:meta.history state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.390814 7f9d1c057700 2 garbage collection: start
2017-10-02 14:52:05.390827 7f9d36440e80 10 cache get: name=default.rgw.log++meta.history : miss
2017-10-02 14:52:05.390868 7f9d1b856700 2 object expiration: start
2017-10-02 14:52:05.390893 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000000
2017-10-02 14:52:05.391901 7f9d36440e80 10 cache put: name=default.rgw.log++meta.history info.flags=0x16
2017-10-02 14:52:05.391962 7f9d36440e80 10 adding default.rgw.log++meta.history to cache LRU end
2017-10-02 14:52:05.391970 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.391977 7f9d36440e80 10 cache get: name=default.rgw.log++meta.history : type miss (requested=0x11, cached=0x16)
2017-10-02 14:52:05.391997 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.399884 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000001
2017-10-02 14:52:05.404821 7f9d36440e80 20 rados->read r=0 bl.length=50
2017-10-02 14:52:05.405046 7f9d36440e80 10 cache put: name=default.rgw.log++meta.history info.flags=0x11
2017-10-02 14:52:05.405057 7f9d36440e80 10 moving default.rgw.log++meta.history to cache LRU end
2017-10-02 14:52:05.405094 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b79f0 obj=.rgw.root:periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.405102 7f9d36440e80 10 cache get: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch : miss
2017-10-02 14:52:05.406049 7f9d36440e80 10 cache put: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch info.flags=0x6
2017-10-02 14:52:05.406070 7f9d36440e80 10 adding .rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch to cache LRU end
2017-10-02 14:52:05.406076 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.406081 7f9d36440e80 10 cache get: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch : type miss (requested=0x1, cached=0x6)
2017-10-02 14:52:05.406083 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.406903 7f9d36440e80 20 rados->read r=0 bl.length=10
2017-10-02 14:52:05.406926 7f9d36440e80 10 cache put: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch info.flags=0x1
2017-10-02 14:52:05.406931 7f9d36440e80 10 moving .rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch to cache LRU end
2017-10-02 14:52:05.406953 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b7a10 obj=.rgw.root:periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 state=0x526954e620 s->prefetch_data=0
2017-10-02 14:52:05.406963 7f9d36440e80 10 cache get: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 : miss
2017-10-02 14:52:05.407773 7f9d36440e80 10 cache put: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 info.flags=0x6
2017-10-02 14:52:05.407792 7f9d36440e80 10 adding .rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 to cache LRU end
2017-10-02 14:52:05.407798 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.407802 7f9d36440e80 10 cache get: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 : type miss (requested=0x1, cached=0x6)
2017-10-02 14:52:05.407806 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.410739 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000002
2017-10-02 14:52:05.419194 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000003
2017-10-02 14:52:05.423216 7f9d36440e80 20 rados->read r=0 bl.length=249
2017-10-02 14:52:05.423240 7f9d36440e80 10 cache put: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 info.flags=0x1
2017-10-02 14:52:05.423245 7f9d36440e80 10 moving .rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 to cache LRU end
2017-10-02 14:52:05.423265 7f9d36440e80 14 rgw period puller: found period 6af5cc56-b57a-4433-a8fd-ab4460098092 in local storage
2017-10-02 14:52:05.424795 7f9d0a834700 20 reqs_thread_entry: start
2017-10-02 14:52:05.425216 7f9d0802f700 20 reqs_thread_entry: start
2017-10-02 14:52:05.425469 7f9d0782e700 20 cr:s=0x526987e000:op=0x526957c800:20MetaMasterTrimPollCR: operate()
2017-10-02 14:52:05.425514 7f9d0782e700 20 run: stack=0x526987e000 is io blocked
2017-10-02 14:52:05.425519 7f9d0782e700 20 cr:s=0x526987e0f0:op=0x5269612700:17DataLogTrimPollCR: operate()
2017-10-02 14:52:05.425535 7f9d0782e700 20 run: stack=0x526987e0f0 is io blocked
2017-10-02 14:52:05.425538 7f9d0682c700 20 reqs_thread_entry: start
2017-10-02 14:52:05.425687 7f9d0582a700 5 schedule life cycle next start time: Mon Oct 2 22:00:00 2017
2017-10-02 14:52:05.425710 7f9d36440e80 20 init_complete bucket index max shards: 0
2017-10-02 14:52:05.425826 7f9d05029700 20 BucketsSyncThread: start
2017-10-02 14:52:05.425887 7f9d04828700 20 UserSyncThread: start
2017-10-02 14:52:05.425899 7f9d04828700 10 ERROR: can't get key: ret=-2
2017-10-02 14:52:05.425902 7f9d04828700 5 ERROR: sync_all_users() returned ret=-2
2017-10-02 14:52:05.425956 7f9d04027700 2 object expiration: start
2017-10-02 14:52:05.425979 7f9d04027700 20 proceeding logshard = reshard.0000000000
2017-10-02 14:52:05.426175 7f9d36440e80 20 RGW hostnames: test.iccluster.epfl.ch
2017-10-02 14:52:05.426189 7f9d36440e80 20 RGW S3website hostnames:
2017-10-02 14:52:05.426552 7f9d03025700 2 keystone revoke thread: start
2017-10-02 14:52:05.426589 7f9d03025700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 14:52:05.428719 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000004
2017-10-02 14:52:05.433484 7f9d04027700 20 proceeding logshard = reshard.0000000001
2017-10-02 14:52:05.434497 7f9d36440e80 0 starting handler: civetweb
2017-10-02 14:52:05.434524 7f9d36440e80 20 civetweb config: canonicalize_url_path: no
2017-10-02 14:52:05.434527 7f9d36440e80 20 civetweb config: decode_url: no
2017-10-02 14:52:05.434529 7f9d36440e80 20 civetweb config: enable_auth_domain_check: no
2017-10-02 14:52:05.434530 7f9d36440e80 20 civetweb config: enable_keep_alive: yes
2017-10-02 14:52:05.434535 7f9d36440e80 20 civetweb config: listening_ports: 127.0.0.1:8080
2017-10-02 14:52:05.434536 7f9d36440e80 20 civetweb config: num_threads: 24
2017-10-02 14:52:05.434538 7f9d36440e80 20 civetweb config: run_as_user: ceph
2017-10-02 14:52:05.434539 7f9d36440e80 20 civetweb config: validate_http_method: no
2017-10-02 14:52:05.437506 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000005
2017-10-02 14:52:05.438111 7f9d36440e80 1 mgrc service_daemon_register rgw.iccluster015 metadata {arch=x86_64,ceph_version=ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable),cpu=Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz,distro=ubuntu,distro_description=Ubuntu 16.04.3 LTS,distro_version=16.04,frontend_config#0=civetweb port=127.0.0.1:8080 num_threads=24,frontend_type#0=civetweb,hostname=iccluster015,kernel_description=#116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017,kernel_version=4.4.0-93-generic,mem_swap_kb=2000056,mem_total_kb=131920952,num_handles=1,os=Linux,pid=41860,zone_id=69d2fd65-fcf9-461b-865f-3dbb053803c4,zone_name=default,zonegroup_id=43d23097-56b9-48a6-ad52-de42341be4bd,zonegroup_name=default}
2017-10-02 14:52:05.438184 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b8710 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x526954f120 s->prefetch_data=0
2017-10-02 14:52:05.438193 7f9d36440e80 10 cache get: name=.rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a : miss
2017-10-02 14:52:05.438858 7f9d36440e80 10 cache put: name=.rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a info.flags=0x6
2017-10-02 14:52:05.438874 7f9d36440e80 10 adding .rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a to cache LRU end
2017-10-02 14:52:05.438880 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.438893 7f9d36440e80 10 cache get: name=.rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a : type miss (requested=0x1, cached=0x6)
2017-10-02 14:52:05.438897 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.439386 7f9d36440e80 20 rados->read r=0 bl.length=107
2017-10-02 14:52:05.439412 7f9d36440e80 10 cache put: name=.rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a info.flags=0x1
2017-10-02 14:52:05.439419 7f9d36440e80 10 moving .rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a to cache LRU end
2017-10-02 14:52:05.439442 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b8880 obj=.rgw.root:periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch state=0x526954f120 s->prefetch_data=0
2017-10-02 14:52:05.439452 7f9d36440e80 10 cache get: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch : miss
2017-10-02 14:52:05.440126 7f9d36440e80 10 cache put: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch info.flags=0x6
2017-10-02 14:52:05.440147 7f9d36440e80 10 adding .rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch to cache LRU end
2017-10-02 14:52:05.440152 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.440162 7f9d36440e80 10 cache get: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch : type miss (requested=0x1, cached=0x6)
2017-10-02 14:52:05.440165 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.440905 7f9d36440e80 20 rados->read r=0 bl.length=10
2017-10-02 14:52:05.440928 7f9d36440e80 10 cache put: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch info.flags=0x1
2017-10-02 14:52:05.440935 7f9d36440e80 10 moving .rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch to cache LRU end
2017-10-02 14:52:05.440980 7f9d36440e80 20 get_system_obj_state: rctx=0x7fff9a0b88a0 obj=.rgw.root:periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 state=0x526954f120 s->prefetch_data=0
2017-10-02 14:52:05.440990 7f9d36440e80 10 cache get: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 : miss
2017-10-02 14:52:05.441698 7f9d36440e80 10 cache put: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 info.flags=0x6
2017-10-02 14:52:05.441712 7f9d36440e80 10 adding .rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 to cache LRU end
2017-10-02 14:52:05.441717 7f9d36440e80 20 get_system_obj_state: s->obj_tag was set empty
2017-10-02 14:52:05.441723 7f9d36440e80 10 cache get: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 : type miss (requested=0x1, cached=0x6)
2017-10-02 14:52:05.441729 7f9d36440e80 20 rados->read ofs=0 len=524288
2017-10-02 14:52:05.442259 7f9d36440e80 20 rados->read r=0 bl.length=846
2017-10-02 14:52:05.442283 7f9d36440e80 10 cache put: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 info.flags=0x1
2017-10-02 14:52:05.442288 7f9d36440e80 10 moving .rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 to cache LRU end
2017-10-02 14:52:05.442324 7f9d36440e80 4 rgw period pusher: No zones to update
2017-10-02 14:52:05.443653 7f9d04027700 20 proceeding logshard = reshard.0000000002
2017-10-02 14:52:05.448001 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000006
2017-10-02 14:52:05.451422 7f9d04027700 20 proceeding logshard = reshard.0000000003
2017-10-02 14:52:05.452476 7f9d36440e80 10 rgw realm watcher: Watching realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a.control
2017-10-02 14:52:05.458217 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000007
2017-10-02 14:52:05.458859 7f9d04027700 20 proceeding logshard = reshard.0000000004
2017-10-02 14:52:05.466536 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000008
2017-10-02 14:52:05.468832 7f9d04027700 20 proceeding logshard = reshard.0000000005
2017-10-02 14:52:05.474224 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000009
2017-10-02 14:52:05.476105 7f9d04027700 20 proceeding logshard = reshard.0000000006
2017-10-02 14:52:05.484104 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000010
2017-10-02 14:52:05.484393 7f9d04027700 20 proceeding logshard = reshard.0000000007
2017-10-02 14:52:05.492802 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000011
2017-10-02 14:52:05.496013 7f9d04027700 20 proceeding logshard = reshard.0000000008
2017-10-02 14:52:05.501852 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000012
2017-10-02 14:52:05.505495 7f9d04027700 20 proceeding logshard = reshard.0000000009
2017-10-02 14:52:05.512160 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000013
2017-10-02 14:52:05.520459 7f9d04027700 20 proceeding logshard = reshard.0000000010
2017-10-02 14:52:05.520725 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000014
2017-10-02 14:52:05.529056 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000015
2017-10-02 14:52:05.532978 7f9d04027700 20 proceeding logshard = reshard.0000000011
2017-10-02 14:52:05.539434 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000016
2017-10-02 14:52:05.545111 7f9d04027700 20 proceeding logshard = reshard.0000000012
2017-10-02 14:52:05.548289 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000017
2017-10-02 14:52:05.553894 7f9d04027700 20 proceeding logshard = reshard.0000000013
2017-10-02 14:52:05.559190 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000018
2017-10-02 14:52:05.564344 7f9d04027700 20 proceeding logshard = reshard.0000000014
2017-10-02 14:52:05.569061 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000019
2017-10-02 14:52:05.572456 7f9d04027700 20 proceeding logshard = reshard.0000000015
2017-10-02 14:52:05.577441 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000020
2017-10-02 14:52:05.583337 7f9d04027700 2 object expiration: stop
2017-10-02 14:52:05.590159 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000021
2017-10-02 14:52:05.597616 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000022
2017-10-02 14:52:05.605611 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000023
2017-10-02 14:52:05.613922 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000024
2017-10-02 14:52:05.623773 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000025
2017-10-02 14:52:05.633008 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000026
2017-10-02 14:52:05.643234 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000027
2017-10-02 14:52:05.651545 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000028
2017-10-02 14:52:05.662239 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000029
2017-10-02 14:52:05.672012 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000030
2017-10-02 14:52:05.679613 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000031
2017-10-02 14:52:05.681137 7f9d1c057700 2 garbage collection: stop
2017-10-02 14:52:05.688434 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000032
2017-10-02 14:52:05.695659 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000033
2017-10-02 14:52:05.704298 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000034
2017-10-02 14:52:05.712648 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000035
2017-10-02 14:52:05.721419 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000036
2017-10-02 14:52:05.728148 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000037
2017-10-02 14:52:05.736315 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000038
2017-10-02 14:52:05.745061 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000039
2017-10-02 14:52:05.754143 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000040
2017-10-02 14:52:05.761713 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000041
2017-10-02 14:52:05.770654 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000042
2017-10-02 14:52:05.779462 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000043
2017-10-02 14:52:05.788883 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000044
2017-10-02 14:52:05.797788 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000045
2017-10-02 14:52:05.805085 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000046
2017-10-02 14:52:05.813512 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000047
2017-10-02 14:52:05.821925 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000048
2017-10-02 14:52:05.835262 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000049
2017-10-02 14:52:05.844025 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000050
2017-10-02 14:52:05.853219 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000051
2017-10-02 14:52:05.861120 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000052
2017-10-02 14:52:05.868953 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000053
2017-10-02 14:52:05.877296 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000054
2017-10-02 14:52:05.885886 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000055
2017-10-02 14:52:05.894889 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000056
2017-10-02 14:52:05.903771 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000057
2017-10-02 14:52:05.912850 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000058
2017-10-02 14:52:05.921945 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000059
2017-10-02 14:52:05.929943 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000060
2017-10-02 14:52:05.939435 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000061
2017-10-02 14:52:05.947943 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000062
2017-10-02 14:52:05.956632 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000063
2017-10-02 14:52:05.963767 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000064
2017-10-02 14:52:05.972682 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000065
2017-10-02 14:52:05.980825 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000066
2017-10-02 14:52:05.989214 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000067
2017-10-02 14:52:05.997787 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000068
2017-10-02 14:52:06.004316 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000069
2017-10-02 14:52:06.017737 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000070
2017-10-02 14:52:06.024722 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000071
2017-10-02 14:52:06.033484 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000072
2017-10-02 14:52:06.042924 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000073
2017-10-02 14:52:06.050762 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000074
2017-10-02 14:52:06.059014 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000075
2017-10-02 14:52:06.070911 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000076
2017-10-02 14:52:06.081822 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000077
2017-10-02 14:52:06.090119 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000078
2017-10-02 14:52:06.098089 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000079
2017-10-02 14:52:06.107313 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000080
2017-10-02 14:52:06.116789 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000081
2017-10-02 14:52:06.125319 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000082
2017-10-02 14:52:06.132964 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000083
2017-10-02 14:52:06.140141 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000084
2017-10-02 14:52:06.149153 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000085
2017-10-02 14:52:06.156967 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000086
2017-10-02 14:52:06.165206 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000087
2017-10-02 14:52:06.173229 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000088
2017-10-02 14:52:06.182173 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000089
2017-10-02 14:52:06.190501 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000090
2017-10-02 14:52:06.199176 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000091
2017-10-02 14:52:06.207825 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000092
2017-10-02 14:52:06.216588 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000093
2017-10-02 14:52:06.224832 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000094
2017-10-02 14:52:06.233876 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000095
2017-10-02 14:52:06.242884 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000096
2017-10-02 14:52:06.251443 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000097
2017-10-02 14:52:06.260148 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000098
2017-10-02 14:52:06.267570 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000099
2017-10-02 14:52:06.276308 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000100
2017-10-02 14:52:06.284940 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000101
2017-10-02 14:52:06.293510 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000102
2017-10-02 14:52:06.301514 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000103
2017-10-02 14:52:06.309691 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000104
2017-10-02 14:52:06.316789 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000105
2017-10-02 14:52:06.324179 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000106
2017-10-02 14:52:06.332826 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000107
2017-10-02 14:52:06.340636 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000108
2017-10-02 14:52:06.349616 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000109
2017-10-02 14:52:06.358929 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000110
2017-10-02 14:52:06.367874 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000111
2017-10-02 14:52:06.376681 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000112
2017-10-02 14:52:06.386598 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000113
2017-10-02 14:52:06.396213 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000114
2017-10-02 14:52:06.405057 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000115
2017-10-02 14:52:06.413705 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000116
2017-10-02 14:52:06.422573 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000117
2017-10-02 14:52:06.430655 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000118
2017-10-02 14:52:06.438495 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000119
2017-10-02 14:52:06.446780 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000120
2017-10-02 14:52:06.455535 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000121
2017-10-02 14:52:06.463450 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000122
2017-10-02 14:52:06.472037 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000123
2017-10-02 14:52:06.480718 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000124
2017-10-02 14:52:06.489403 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000125
2017-10-02 14:52:06.498468 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000126
2017-10-02 14:52:06.506544 7f9d1b856700 2 object expiration: stop
2017-10-02 14:52:27.014552 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:52:49.014675 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:53:11.014797 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:53:33.014921 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:53:55.015021 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:54:17.015143 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:54:21.264087 7f9d1d85a700 10 RGWWatcher::handle_notify() notify_id 11918534246408 cookie 353953183232 notifier 266445 bl.length()=695
2017-10-02 14:54:21.264153 7f9d1d85a700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 info.flags=0x17
2017-10-02 14:54:21.264165 7f9d1d85a700 10 adding default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 to cache LRU end
2017-10-02 14:54:21.264170 7f9d1d85a700 10 updating xattr: name=user.rgw.acl bl.length()=163
2017-10-02 14:54:21.264172 7f9d1d85a700 10 updating xattr: name=user.rgw.idtag bl.length()=0
2017-10-02 14:54:21.393096 7f9d1d85a700 10 RGWWatcher::handle_notify() notify_id 11918534246409 cookie 353953183232 notifier 266445 bl.length()=764
2017-10-02 14:54:21.393134 7f9d1d85a700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266391.1 info.flags=0x17
2017-10-02 14:54:21.393146 7f9d1d85a700 10 adding default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266391.1 to cache LRU end
2017-10-02 14:54:21.393151 7f9d1d85a700 10 updating xattr: name=user.rgw.acl bl.length()=163
2017-10-02 14:54:21.393153 7f9d1d85a700 10 updating xattr: name=user.rgw.idtag bl.length()=0
2017-10-02 14:54:39.015263 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:55:01.015390 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:55:23.015523 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:55:45.015645 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:56:07.015744 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:56:29.015871 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:56:51.015992 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:57:13.016119 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:57:35.016207 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:57:57.016323 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:58:19.016423 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:58:41.016516 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:59:03.016634 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:59:25.016764 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 14:59:47.016861 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:00:09.016983 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:00:31.017109 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:00:53.017232 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:01:15.017359 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:01:37.017452 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:01:59.017573 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:02:05.506661 7f9d1b856700 2 object expiration: start
2017-10-02 15:02:05.506681 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000000
2017-10-02 15:02:05.515791 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000001
2017-10-02 15:02:05.523709 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000002
2017-10-02 15:02:05.532211 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000003
2017-10-02 15:02:05.539482 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000004
2017-10-02 15:02:05.546664 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000005
2017-10-02 15:02:05.561432 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000006
2017-10-02 15:02:05.569705 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000007
2017-10-02 15:02:05.578712 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000008
2017-10-02 15:02:05.583421 7f9d04027700 2 object expiration: start
2017-10-02 15:02:05.583437 7f9d04027700 20 proceeding logshard = reshard.0000000000
2017-10-02 15:02:05.586844 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000009
2017-10-02 15:02:05.594147 7f9d04027700 20 proceeding logshard = reshard.0000000001
2017-10-02 15:02:05.595054 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000010
2017-10-02 15:02:05.604043 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000011
2017-10-02 15:02:05.604519 7f9d04027700 20 proceeding logshard = reshard.0000000002
2017-10-02 15:02:05.612052 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000012
2017-10-02 15:02:05.612056 7f9d04027700 20 proceeding logshard = reshard.0000000003
2017-10-02 15:02:05.620623 7f9d04027700 20 proceeding logshard = reshard.0000000004
2017-10-02 15:02:05.620743 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000013
2017-10-02 15:02:05.629742 7f9d04027700 20 proceeding logshard = reshard.0000000005
2017-10-02 15:02:05.631805 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000014
2017-10-02 15:02:05.637027 7f9d04027700 20 proceeding logshard = reshard.0000000006
2017-10-02 15:02:05.640389 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000015
2017-10-02 15:02:05.644755 7f9d04027700 20 proceeding logshard = reshard.0000000007
2017-10-02 15:02:05.652450 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000016
2017-10-02 15:02:05.653131 7f9d04027700 20 proceeding logshard = reshard.0000000008
2017-10-02 15:02:05.661716 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000017
2017-10-02 15:02:05.661737 7f9d04027700 20 proceeding logshard = reshard.0000000009
2017-10-02 15:02:05.671666 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000018
2017-10-02 15:02:05.671813 7f9d04027700 20 proceeding logshard = reshard.0000000010
2017-10-02 15:02:05.680024 7f9d04027700 20 proceeding logshard = reshard.0000000011
2017-10-02 15:02:05.682675 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000019
2017-10-02 15:02:05.686891 7f9d04027700 20 proceeding logshard = reshard.0000000012
2017-10-02 15:02:05.691155 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000020
2017-10-02 15:02:05.697113 7f9d04027700 20 proceeding logshard = reshard.0000000013
2017-10-02 15:02:05.699729 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000021
2017-10-02 15:02:05.707198 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000022
2017-10-02 15:02:05.707617 7f9d04027700 20 proceeding logshard = reshard.0000000014
2017-10-02 15:02:05.716840 7f9d04027700 20 proceeding logshard = reshard.0000000015
2017-10-02 15:02:05.719432 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000023
2017-10-02 15:02:05.724747 7f9d04027700 2 object expiration: stop
2017-10-02 15:02:05.727998 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000024
2017-10-02 15:02:05.736890 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000025
2017-10-02 15:02:05.746374 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000026
2017-10-02 15:02:05.756025 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000027
2017-10-02 15:02:05.763025 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000028
2017-10-02 15:02:05.772914 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000029
2017-10-02 15:02:05.780285 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000030
2017-10-02 15:02:05.787842 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000031
2017-10-02 15:02:05.795678 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000032
2017-10-02 15:02:05.803656 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000033
2017-10-02 15:02:05.810132 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000034
2017-10-02 15:02:05.818207 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000035
2017-10-02 15:02:05.825369 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000036
2017-10-02 15:02:05.833293 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000037
2017-10-02 15:02:05.842062 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000038
2017-10-02 15:02:05.849641 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000039
2017-10-02 15:02:05.857316 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000040
2017-10-02 15:02:05.866798 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000041
2017-10-02 15:02:05.880572 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000042
2017-10-02 15:02:05.891327 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000043
2017-10-02 15:02:05.899003 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000044
2017-10-02 15:02:05.908005 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000045
2017-10-02 15:02:05.917632 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000046
2017-10-02 15:02:05.925120 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000047
2017-10-02 15:02:05.933206 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000048
2017-10-02 15:02:05.942355 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000049
2017-10-02 15:02:05.950500 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000050
2017-10-02 15:02:05.958646 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000051
2017-10-02 15:02:05.966874 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000052
2017-10-02 15:02:05.977410 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000053
2017-10-02 15:02:05.984680 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000054
2017-10-02 15:02:05.992511 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000055
2017-10-02 15:02:06.000442 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000056
2017-10-02 15:02:06.009010 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000057
2017-10-02 15:02:06.016116 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000058
2017-10-02 15:02:06.023038 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000059
2017-10-02 15:02:06.030420 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000060
2017-10-02 15:02:06.037934 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000061
2017-10-02 15:02:06.045495 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000062
2017-10-02 15:02:06.054072 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000063
2017-10-02 15:02:06.062071 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000064
2017-10-02 15:02:06.070805 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000065
2017-10-02 15:02:06.077905 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000066
2017-10-02 15:02:06.085638 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000067
2017-10-02 15:02:06.093143 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000068
2017-10-02 15:02:06.101782 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000069
2017-10-02 15:02:06.109200 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000070
2017-10-02 15:02:06.137529 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000071
2017-10-02 15:02:06.144567 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000072
2017-10-02 15:02:06.152222 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000073
2017-10-02 15:02:06.161291 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000074
2017-10-02 15:02:06.168494 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000075
2017-10-02 15:02:06.174921 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000076
2017-10-02 15:02:06.183446 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000077
2017-10-02 15:02:06.190992 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000078
2017-10-02 15:02:06.199706 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000079
2017-10-02 15:02:06.210049 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000080
2017-10-02 15:02:06.217694 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000081
2017-10-02 15:02:06.225168 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000082
2017-10-02 15:02:06.233496 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000083
2017-10-02 15:02:06.240472 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000084
2017-10-02 15:02:06.247632 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000085
2017-10-02 15:02:06.254463 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000086
2017-10-02 15:02:06.266940 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000087
2017-10-02 15:02:06.275706 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000088
2017-10-02 15:02:06.285844 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000089
2017-10-02 15:02:06.293719 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000090
2017-10-02 15:02:06.303228 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000091
2017-10-02 15:02:06.314707 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000092
2017-10-02 15:02:06.322307 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000093
2017-10-02 15:02:06.330100 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000094
2017-10-02 15:02:06.338824 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000095
2017-10-02 15:02:06.347068 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000096
2017-10-02 15:02:06.354011 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000097
2017-10-02 15:02:06.360330 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000098
2017-10-02 15:02:06.369871 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000099
2017-10-02 15:02:06.377576 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000100
2017-10-02 15:02:06.384138 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000101
2017-10-02 15:02:06.392119 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000102
2017-10-02 15:02:06.401556 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000103
2017-10-02 15:02:06.410101 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000104
2017-10-02 15:02:06.419868 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000105
2017-10-02 15:02:06.427392 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000106
2017-10-02 15:02:06.437355 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000107
2017-10-02 15:02:06.446729 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000108
2017-10-02 15:02:06.455756 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000109
2017-10-02 15:02:06.463388 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000110
2017-10-02 15:02:06.472209 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000111
2017-10-02 15:02:06.481757 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000112
2017-10-02 15:02:06.489521 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000113
2017-10-02 15:02:06.497838 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000114
2017-10-02 15:02:06.506058 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000115
2017-10-02 15:02:06.515062 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000116
2017-10-02 15:02:06.524038 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000117
2017-10-02 15:02:06.532229 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000118
2017-10-02 15:02:06.539247 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000119
2017-10-02 15:02:06.547652 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000120
2017-10-02 15:02:06.554860 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000121
2017-10-02 15:02:06.561422 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000122
2017-10-02 15:02:06.568631 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000123
2017-10-02 15:02:06.576092 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000124
2017-10-02 15:02:06.583106 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000125
2017-10-02 15:02:06.591326 7f9d1b856700 20 proceeding shard = obj_delete_at_hint.0000000126
2017-10-02 15:02:06.598255 7f9d1b856700 2 object expiration: stop
2017-10-02 15:02:21.017662 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:02:43.017797 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:03:05.017940 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:03:27.018069 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:03:49.018212 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:04:11.018338 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:04:33.018478 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:04:55.018602 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:05:17.018692 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:05:39.018835 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:06:01.018958 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:06:23.019080 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:06:45.019202 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:07:05.426731 7f9d03025700 2 keystone revoke thread: start
2017-10-02 15:07:05.426754 7f9d03025700 0 ERROR: keystone revocation processing returned error r=-22
2017-10-02 15:07:07.019329 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:07:26.960386 7f9d1d85a700 10 RGWWatcher::handle_notify() notify_id 11918534246410 cookie 353953183232 notifier 266445 bl.length()=695
2017-10-02 15:07:26.960434 7f9d1d85a700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 info.flags=0x17
2017-10-02 15:07:26.960441 7f9d1d85a700 10 moving default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 to cache LRU end
2017-10-02 15:07:26.960450 7f9d1d85a700 10 updating xattr: name=user.rgw.acl bl.length()=163
2017-10-02 15:07:26.960454 7f9d1d85a700 10 updating xattr: name=user.rgw.idtag bl.length()=0
2017-10-02 15:07:26.966388 7f9d1d85a700 10 RGWWatcher::handle_notify() notify_id 11918534246408 cookie 353953179776 notifier 266445 bl.length()=419
2017-10-02 15:07:26.966425 7f9d1d85a700 10 cache put: name=default.rgw.meta+root+image-net info.flags=0x18
2017-10-02 15:07:26.966434 7f9d1d85a700 10 adding default.rgw.meta+root+image-net to cache LRU end
2017-10-02 15:07:26.966437 7f9d1d85a700 10 appending xattr: name=user.rgw.acl bl.length()=246
2017-10-02 15:07:26.973853 7f9d1d85a700 10 RGWWatcher::handle_notify() notify_id 11918534246411 cookie 353953183232 notifier 266445 bl.length()=395
2017-10-02 15:07:26.973887 7f9d1d85a700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 info.flags=0x18
2017-10-02 15:07:26.973896 7f9d1d85a700 10 moving default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 to cache LRU end
2017-10-02 15:07:26.973900 7f9d1d85a700 10 appending xattr: name=user.rgw.acl bl.length()=163
2017-10-02 15:07:26.987645 7f9d1d85a700 10 RGWWatcher::handle_notify() notify_id 11918534246409 cookie 353953179776 notifier 266445 bl.length()=619
2017-10-02 15:07:26.987683 7f9d1d85a700 10 cache put: name=default.rgw.meta+root+image-net info.flags=0x17
2017-10-02 15:07:26.987692 7f9d1d85a700 10 moving default.rgw.meta+root+image-net to cache LRU end
2017-10-02 15:07:26.987698 7f9d1d85a700 10 updating xattr: name=user.rgw.acl bl.length()=246
2017-10-02 15:07:26.987702 7f9d1d85a700 10 updating xattr: name=user.rgw.idtag bl.length()=0
2017-10-02 15:07:26.994228 7f9d1d85a700 10 RGWWatcher::handle_notify() notify_id 11918534246412 cookie 353953183232 notifier 266445 bl.length()=764
2017-10-02 15:07:26.994262 7f9d1d85a700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266391.1 info.flags=0x17
2017-10-02 15:07:26.994271 7f9d1d85a700 10 moving default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266391.1 to cache LRU end
2017-10-02 15:07:26.994278 7f9d1d85a700 10 updating xattr: name=user.rgw.acl bl.length()=163
2017-10-02 15:07:26.994282 7f9d1d85a700 10 updating xattr: name=user.rgw.idtag bl.length()=0
2017-10-02 15:07:29.019470 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:07:51.019606 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:08:13.019753 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:08:35.019871 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:08:57.019986 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:09:19.020101 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:09:41.020218 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:10:03.020343 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:10:25.020478 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-10-02 15:10:47.020607 7f9d1d059700 2 RGWDataChangesLog::ChangesRenewThread: start
(2-2/3)