luminous-rgw-iccluster023.log - rgw - Ceph

Bug #21619 » luminous-rgw-iccluster023.log

Radosgw debug log #2 - Yoann Moulin, 10/02/2017 01:32 PM

 -10-02 06:25:01.408719 7fd9a84ba700  0 ERROR: failed to clone shard, completion_mgr.get_next() returned ret=-125
 -10-02 06:25:01.447135 7fd998293700  1 rgw realm reloader: Store closed
 -10-02 06:25:01.998443 7fd998293700  1 rgw realm reloader: Creating new store
 -10-02 06:25:01.998738 7fd998293700  1 mgrc service_daemon_register rgw.iccluster023 metadata {arch=x86_64,ceph_version=ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable),cpu=Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz,distro=ubuntu,distro_description=Ubuntu 16.04.3 LTS,distro_version=16.04,frontend_config#0=civetweb port=127.0.0.1:8080 num_threads=24,frontend_type#0=civetweb,hostname=iccluster023,kernel_description=#116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017,kernel_version=4.4.0-93-generic,mem_swap_kb=2000056,mem_total_kb=131920952,num_handles=1,os=Linux,pid=2891,zone_id=69d2fd65-fcf9-461b-865f-3dbb053803c4,zone_name=default,zonegroup_id=43d23097-56b9-48a6-ad52-de42341be4bd,zonegroup_name=default}
 -10-02 06:25:01.998790 7fd998293700  1 rgw realm reloader: Finishing initialization of new store
 -10-02 06:25:01.998793 7fd998293700  1 rgw realm reloader:  - REST subsystem init
 -10-02 06:25:01.998823 7fd998293700  1 rgw realm reloader:  - user subsystem init
 -10-02 06:25:01.998831 7fd998293700  1 rgw realm reloader:  - user subsystem init
 -10-02 06:25:01.998838 7fd998293700  1 rgw realm reloader:  - usage subsystem init
 -10-02 06:25:01.998911 7fd998293700  1 rgw realm reloader: Resuming frontends with new realm configuration.
 -10-02 06:36:58.428171 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 06:51:58.428271 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 07:06:58.428362 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 07:21:58.428458 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 07:36:58.428520 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 07:51:58.428652 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 08:06:58.428756 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 08:21:58.428866 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 08:36:58.428973 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 08:51:58.429090 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 09:06:58.429192 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 09:21:58.429301 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 09:36:58.429427 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 09:51:58.429542 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 10:06:58.429656 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 10:21:58.429764 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 10:36:58.429880 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 10:51:58.430007 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 11:06:58.430097 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 11:21:58.430242 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 11:36:58.430393 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 11:51:58.430527 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 12:06:58.430651 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 12:21:58.430768 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 12:36:58.430906 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 12:51:58.431033 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 13:06:58.431125 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 13:21:58.431257 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 13:36:58.431380 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 13:51:58.431483 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 14:06:58.431616 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 14:21:58.431748 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 14:36:58.431900 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 14:51:58.432022 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 14:52:39.101815 7fd9a52ad700 -1 received  signal: Terminated from  PID: 1 task name: /sbin/init splash  UID: 0
 -10-02 14:52:39.101834 7fd9a52ad700  1 handle_sigterm
 -10-02 14:52:39.101887 7fd9a52ad700  1 handle_sigterm set alarm for 120
 -10-02 14:52:39.101886 7fd9d68cbe80 -1 shutting down
 -10-02 14:52:39.132678 7fd9a84ba700  0 ERROR: failed to clone shard, completion_mgr.get_next() returned ret=-125
 -10-02 14:52:39.171645 7fd9d68cbe80  1 final shutdown
 -10-02 14:52:39.283328 7fceb68d3e80  0 deferred set uid:gid to 64045:64045 (ceph:ceph)
 -10-02 14:52:39.283374 7fceb68d3e80  0 ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable), process (unknown), pid 33119
 -10-02 14:52:39.291750 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39da0 obj=.rgw.root:default.realm state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.291786 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:52:39.294025 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.294044 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.294611 7fceb68d3e80 20 rados->read r=0 bl.length=46
 -10-02 14:52:39.294664 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39e20 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.296413 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.296430 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.297295 7fceb68d3e80 20 rados->read r=0 bl.length=107
 -10-02 14:52:39.297348 7fceb68d3e80 20 realm  default b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
 -10-02 14:52:39.297370 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39c10 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.298182 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.298199 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.298978 7fceb68d3e80 20 rados->read r=0 bl.length=107
 -10-02 14:52:39.299018 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39d80 obj=.rgw.root:periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.300542 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.300558 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.301275 7fceb68d3e80 20 rados->read r=0 bl.length=10
 -10-02 14:52:39.301319 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39da0 obj=.rgw.root:periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.302143 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.302162 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.302967 7fceb68d3e80 20 rados->read r=0 bl.length=846
 -10-02 14:52:39.303034 7fceb68d3e80 20 current period e7bfcb5a-829b-418f-ae26-d6573a5cc8b9
 -10-02 14:52:39.303051 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c394f0 obj=.rgw.root:converted state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.304666 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38940 obj=.rgw.root:default.realm state=0x32ada82e60 s->prefetch_data=0
 -10-02 14:52:39.305594 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.305613 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.306417 7fceb68d3e80 20 rados->read r=0 bl.length=46
 -10-02 14:52:39.306463 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c389c0 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82e60 s->prefetch_data=0
 -10-02 14:52:39.307343 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.307361 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.308133 7fceb68d3e80 20 rados->read r=0 bl.length=107
 -10-02 14:52:39.308161 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38ad0 obj=.rgw.root:default.region state=0x32ada82e60 s->prefetch_data=0
 -10-02 14:52:39.557156 7fceb68d3e80 20 RGWRados::pool_iterate: got zonegroup_info.43d23097-56b9-48a6-ad52-de42341be4bd
 -10-02 14:52:39.557176 7fceb68d3e80 20 RGWRados::pool_iterate: got periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch
 -10-02 14:52:39.557180 7fceb68d3e80 20 RGWRados::pool_iterate: got periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1
 -10-02 14:52:39.557182 7fceb68d3e80 20 RGWRados::pool_iterate: got realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
 -10-02 14:52:39.557186 7fceb68d3e80 20 RGWRados::pool_iterate: got default.realm
 -10-02 14:52:39.557188 7fceb68d3e80 20 RGWRados::pool_iterate: got default.zone.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
 -10-02 14:52:39.557210 7fceb68d3e80 20 RGWRados::pool_iterate: got periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch
 -10-02 14:52:39.557213 7fceb68d3e80 20 RGWRados::pool_iterate: got default.zonegroup.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
 -10-02 14:52:39.557215 7fceb68d3e80 20 RGWRados::pool_iterate: got periods.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a:staging
 -10-02 14:52:39.557216 7fceb68d3e80 20 RGWRados::pool_iterate: got realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a.control
 -10-02 14:52:39.557218 7fceb68d3e80 20 RGWRados::pool_iterate: got period_config.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
 -10-02 14:52:39.557219 7fceb68d3e80 20 RGWRados::pool_iterate: got zone_info.69d2fd65-fcf9-461b-865f-3dbb053803c4
 -10-02 14:52:39.557241 7fceb68d3e80 20 RGWRados::pool_iterate: got realms_names.default
 -10-02 14:52:39.557244 7fceb68d3e80 20 RGWRados::pool_iterate: got zone_names.default
 -10-02 14:52:39.557246 7fceb68d3e80 20 RGWRados::pool_iterate: got zonegroups_names.default
 -10-02 14:52:39.557247 7fceb68d3e80 20 RGWRados::pool_iterate: got periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1
 -10-02 14:52:39.557268 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38bc0 obj=.rgw.root:zone_names.default state=0x32ada82e60 s->prefetch_data=0
 -10-02 14:52:39.558064 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.558083 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.558704 7fceb68d3e80 20 rados->read r=0 bl.length=46
 -10-02 14:52:39.558764 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38bc0 obj=.rgw.root:zone_info.69d2fd65-fcf9-461b-865f-3dbb053803c4 state=0x32ada82e60 s->prefetch_data=0
 -10-02 14:52:39.560437 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.560455 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.561322 7fceb68d3e80 20 rados->read r=0 bl.length=724
 -10-02 14:52:39.561375 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38bc0 obj=.rgw.root:zonegroups_names.default state=0x32ada82e60 s->prefetch_data=0
 -10-02 14:52:39.562122 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.562140 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.562761 7fceb68d3e80 20 rados->read r=0 bl.length=46
 -10-02 14:52:39.562797 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38bc0 obj=.rgw.root:zonegroup_info.43d23097-56b9-48a6-ad52-de42341be4bd state=0x32ada82e60 s->prefetch_data=0
 -10-02 14:52:39.563629 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.563645 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.564418 7fceb68d3e80 20 rados->read r=0 bl.length=369
 -10-02 14:52:39.564467 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39fb0 obj=.rgw.root:region_map state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.565318 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39630 obj=.rgw.root:default.realm state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.566464 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.566481 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.567298 7fceb68d3e80 20 rados->read r=0 bl.length=46
 -10-02 14:52:39.567335 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c396b0 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.568256 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.568274 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.569137 7fceb68d3e80 20 rados->read r=0 bl.length=107
 -10-02 14:52:39.569184 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c397c0 obj=.rgw.root:default.zonegroup.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.570143 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.570161 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.571006 7fceb68d3e80 20 rados->read r=0 bl.length=46
 -10-02 14:52:39.571047 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39a40 obj=.rgw.root:zonegroup_info.43d23097-56b9-48a6-ad52-de42341be4bd state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.571888 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.571906 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.572636 7fceb68d3e80 20 rados->read r=0 bl.length=369
 -10-02 14:52:39.572675 7fceb68d3e80 20 period zonegroup init ret 0
 -10-02 14:52:39.572678 7fceb68d3e80 20 period zonegroup name default
 -10-02 14:52:39.572681 7fceb68d3e80 20 using current period zonegroup default
 -10-02 14:52:39.572692 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39630 obj=.rgw.root:default.realm state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.576128 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.576144 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.577040 7fceb68d3e80 20 rados->read r=0 bl.length=46
 -10-02 14:52:39.577077 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c396b0 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.577993 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.578012 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.578807 7fceb68d3e80 20 rados->read r=0 bl.length=107
 -10-02 14:52:39.578847 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c397c0 obj=.rgw.root:default.zone.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.579606 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.579623 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.580350 7fceb68d3e80 20 rados->read r=0 bl.length=46
 -10-02 14:52:39.580385 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39a40 obj=.rgw.root:zone_info.69d2fd65-fcf9-461b-865f-3dbb053803c4 state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.581242 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.581259 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.582137 7fceb68d3e80 20 rados->read r=0 bl.length=724
 -10-02 14:52:39.582190 7fceb68d3e80 10 Cannot find current period zone using local zone
 -10-02 14:52:39.582202 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39e20 obj=.rgw.root:zone_info.69d2fd65-fcf9-461b-865f-3dbb053803c4 state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.583006 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.583023 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.583809 7fceb68d3e80 20 rados->read r=0 bl.length=724
 -10-02 14:52:39.583870 7fceb68d3e80 20 zone default
 -10-02 14:52:39.592370 7fceb68d3e80 20 add_watcher() i=0
 -10-02 14:52:39.599381 7fceb68d3e80 20 add_watcher() i=1
 -10-02 14:52:39.607364 7fceb68d3e80 20 add_watcher() i=2
 -10-02 14:52:39.614888 7fceb68d3e80 20 add_watcher() i=3
 -10-02 14:52:39.622705 7fceb68d3e80 20 add_watcher() i=4
 -10-02 14:52:39.630970 7fceb68d3e80 20 add_watcher() i=5
 -10-02 14:52:39.639366 7fceb68d3e80 20 add_watcher() i=6
 -10-02 14:52:39.646629 7fceb68d3e80 20 add_watcher() i=7
 -10-02 14:52:39.646646 7fceb68d3e80  2 all 8 watchers are set, enabling cache
 -10-02 14:52:39.646796 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39730 obj=default.rgw.log:meta.history state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.646798 7fce9c4ea700  2 garbage collection: start
 -10-02 14:52:39.646809 7fceb68d3e80 10 cache get: name=default.rgw.log++meta.history : miss
 -10-02 14:52:39.647026 7fce9bce9700  2 object expiration: start
 -10-02 14:52:39.647066 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000000
 -10-02 14:52:39.647679 7fceb68d3e80 10 cache put: name=default.rgw.log++meta.history info.flags=0x16
 -10-02 14:52:39.647726 7fceb68d3e80 10 adding default.rgw.log++meta.history to cache LRU end
 -10-02 14:52:39.647735 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.647742 7fceb68d3e80 10 cache get: name=default.rgw.log++meta.history : type miss (requested=0x11, cached=0x16)
 -10-02 14:52:39.647762 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.648802 7fceb68d3e80 20 rados->read r=0 bl.length=50
 -10-02 14:52:39.648825 7fceb68d3e80 10 cache put: name=default.rgw.log++meta.history info.flags=0x11
 -10-02 14:52:39.648829 7fceb68d3e80 10 moving default.rgw.log++meta.history to cache LRU end
 -10-02 14:52:39.648867 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38f50 obj=.rgw.root:periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.648880 7fceb68d3e80 10 cache get: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch : miss
 -10-02 14:52:39.649777 7fceb68d3e80 10 cache put: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch info.flags=0x6
 -10-02 14:52:39.649797 7fceb68d3e80 10 adding .rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch to cache LRU end
 -10-02 14:52:39.649803 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.649807 7fceb68d3e80 10 cache get: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch : type miss (requested=0x1, cached=0x6)
 -10-02 14:52:39.649810 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.650596 7fceb68d3e80 20 rados->read r=0 bl.length=10
 -10-02 14:52:39.650621 7fceb68d3e80 10 cache put: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch info.flags=0x1
 -10-02 14:52:39.650625 7fceb68d3e80 10 moving .rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch to cache LRU end
 -10-02 14:52:39.650648 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38f70 obj=.rgw.root:periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 state=0x32ada82620 s->prefetch_data=0
 -10-02 14:52:39.650658 7fceb68d3e80 10 cache get: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 : miss
 -10-02 14:52:39.651586 7fceb68d3e80 10 cache put: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 info.flags=0x6
 -10-02 14:52:39.651607 7fceb68d3e80 10 adding .rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 to cache LRU end
 -10-02 14:52:39.651614 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.651620 7fceb68d3e80 10 cache get: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 : type miss (requested=0x1, cached=0x6)
 -10-02 14:52:39.651623 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.652764 7fceb68d3e80 20 rados->read r=0 bl.length=249
 -10-02 14:52:39.653017 7fceb68d3e80 10 cache put: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 info.flags=0x1
 -10-02 14:52:39.653028 7fceb68d3e80 10 moving .rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 to cache LRU end
 -10-02 14:52:39.653048 7fceb68d3e80 14 rgw period puller: found period 6af5cc56-b57a-4433-a8fd-ab4460098092 in local storage
 -10-02 14:52:39.654517 7fce8acc7700 20 reqs_thread_entry: start
 -10-02 14:52:39.655007 7fce884c2700 20 reqs_thread_entry: start
 -10-02 14:52:39.655201 7fce87cc1700 20 cr:s=0x32add86000:op=0x32adab0800:20MetaMasterTrimPollCR: operate()
 -10-02 14:52:39.655246 7fce87cc1700 20 run: stack=0x32add86000 is io blocked
 -10-02 14:52:39.655255 7fce87cc1700 20 cr:s=0x32add860f0:op=0x32adb46700:17DataLogTrimPollCR: operate()
 -10-02 14:52:39.655274 7fce87cc1700 20 run: stack=0x32add860f0 is io blocked
 -10-02 14:52:39.655322 7fce86cbf700 20 reqs_thread_entry: start
 -10-02 14:52:39.655383 7fce854bc700 20 BucketsSyncThread: start
 -10-02 14:52:39.655433 7fceb68d3e80 20 init_complete bucket index max shards: 0
 -10-02 14:52:39.655447 7fce84cbb700 20 UserSyncThread: start
 -10-02 14:52:39.655431 7fce85cbd700  5 schedule life cycle next start time: Mon Oct  2 22:00:00 2017
 -10-02 14:52:39.655470 7fce84cbb700 10 ERROR: can't get key: ret=-2
 -10-02 14:52:39.655477 7fce84cbb700  5 ERROR: sync_all_users() returned ret=-2
 -10-02 14:52:39.655500 7fce844ba700  2 object expiration: start
 -10-02 14:52:39.655515 7fce844ba700 20 proceeding logshard = reshard.0000000000
 -10-02 14:52:39.655704 7fceb68d3e80 20 RGW hostnames: test.iccluster.epfl.ch
 -10-02 14:52:39.655712 7fceb68d3e80 20 RGW S3website hostnames:
 -10-02 14:52:39.655945 7fce834b8700  2 keystone revoke thread: start
 -10-02 14:52:39.655957 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000001
 -10-02 14:52:39.655978 7fce834b8700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 14:52:39.657718 7fceb68d3e80  0 starting handler: civetweb
 -10-02 14:52:39.657744 7fceb68d3e80 20 civetweb config: canonicalize_url_path: no
 -10-02 14:52:39.657747 7fceb68d3e80 20 civetweb config: decode_url: no
 -10-02 14:52:39.657749 7fceb68d3e80 20 civetweb config: enable_auth_domain_check: no
 -10-02 14:52:39.657750 7fceb68d3e80 20 civetweb config: enable_keep_alive: yes
 -10-02 14:52:39.657753 7fceb68d3e80 20 civetweb config: listening_ports: 127.0.0.1:8080
 -10-02 14:52:39.657755 7fceb68d3e80 20 civetweb config: num_threads: 24
 -10-02 14:52:39.657757 7fceb68d3e80 20 civetweb config: run_as_user: ceph
 -10-02 14:52:39.657758 7fceb68d3e80 20 civetweb config: validate_http_method: no
 -10-02 14:52:39.660368 7fceb68d3e80  1 mgrc service_daemon_register rgw.iccluster023 metadata {arch=x86_64,ceph_version=ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable),cpu=Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz,distro=ubuntu,distro_description=Ubuntu 16.04.3 LTS,distro_version=16.04,frontend_config#0=civetweb port=127.0.0.1:8080 num_threads=24,frontend_type#0=civetweb,hostname=iccluster023,kernel_description=#116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017,kernel_version=4.4.0-93-generic,mem_swap_kb=2000056,mem_total_kb=131920952,num_handles=1,os=Linux,pid=33119,zone_id=69d2fd65-fcf9-461b-865f-3dbb053803c4,zone_name=default,zonegroup_id=43d23097-56b9-48a6-ad52-de42341be4bd,zonegroup_name=default}
 -10-02 14:52:39.660435 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39c70 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada83120 s->prefetch_data=0
 -10-02 14:52:39.660446 7fceb68d3e80 10 cache get: name=.rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a : miss
 -10-02 14:52:39.661236 7fceb68d3e80 10 cache put: name=.rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a info.flags=0x6
 -10-02 14:52:39.661252 7fceb68d3e80 10 adding .rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a to cache LRU end
 -10-02 14:52:39.661257 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.661262 7fceb68d3e80 10 cache get: name=.rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a : type miss (requested=0x1, cached=0x6)
 -10-02 14:52:39.661265 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.662032 7fceb68d3e80 20 rados->read r=0 bl.length=107
 -10-02 14:52:39.662045 7fceb68d3e80 10 cache put: name=.rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a info.flags=0x1
 -10-02 14:52:39.662050 7fceb68d3e80 10 moving .rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a to cache LRU end
 -10-02 14:52:39.662069 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39de0 obj=.rgw.root:periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch state=0x32ada83120 s->prefetch_data=0
 -10-02 14:52:39.662086 7fceb68d3e80 10 cache get: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch : miss
 -10-02 14:52:39.662727 7fceb68d3e80 10 cache put: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch info.flags=0x6
 -10-02 14:52:39.662740 7fceb68d3e80 10 adding .rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch to cache LRU end
 -10-02 14:52:39.662744 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.662747 7fceb68d3e80 10 cache get: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch : type miss (requested=0x1, cached=0x6)
 -10-02 14:52:39.662749 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.663236 7fceb68d3e80 20 rados->read r=0 bl.length=10
 -10-02 14:52:39.663253 7fceb68d3e80 10 cache put: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch info.flags=0x1
 -10-02 14:52:39.663256 7fceb68d3e80 10 moving .rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch to cache LRU end
 -10-02 14:52:39.663295 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39e00 obj=.rgw.root:periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 state=0x32ada83120 s->prefetch_data=0
 -10-02 14:52:39.663304 7fceb68d3e80 10 cache get: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 : miss
 -10-02 14:52:39.664021 7fceb68d3e80 10 cache put: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 info.flags=0x6
 -10-02 14:52:39.664033 7fceb68d3e80 10 adding .rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 to cache LRU end
 -10-02 14:52:39.664037 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
 -10-02 14:52:39.664041 7fceb68d3e80 10 cache get: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 : type miss (requested=0x1, cached=0x6)
 -10-02 14:52:39.664043 7fceb68d3e80 20 rados->read ofs=0 len=524288
 -10-02 14:52:39.664742 7fceb68d3e80 20 rados->read r=0 bl.length=846
 -10-02 14:52:39.664756 7fceb68d3e80 10 cache put: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 info.flags=0x1
 -10-02 14:52:39.664759 7fceb68d3e80 10 moving .rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 to cache LRU end
 -10-02 14:52:39.664790 7fceb68d3e80  4 rgw period pusher: No zones to update
 -10-02 14:52:39.667182 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000002
 -10-02 14:52:39.667365 7fce844ba700 20 proceeding logshard = reshard.0000000001
 -10-02 14:52:39.672326 7fceb68d3e80 10 rgw realm watcher: Watching realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a.control
 -10-02 14:52:39.677440 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000003
 -10-02 14:52:39.677801 7fce844ba700 20 proceeding logshard = reshard.0000000002
 -10-02 14:52:39.686557 7fce844ba700 20 proceeding logshard = reshard.0000000003
 -10-02 14:52:39.687589 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000004
 -10-02 14:52:39.694650 7fce844ba700 20 proceeding logshard = reshard.0000000004
 -10-02 14:52:39.704502 7fce844ba700 20 proceeding logshard = reshard.0000000005
 -10-02 14:52:39.706923 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000005
 -10-02 14:52:39.711889 7fce844ba700 20 proceeding logshard = reshard.0000000006
 -10-02 14:52:39.715845 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000006
 -10-02 14:52:39.721913 7fce844ba700 20 proceeding logshard = reshard.0000000007
 -10-02 14:52:39.725995 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000007
 -10-02 14:52:39.730904 7fce844ba700 20 proceeding logshard = reshard.0000000008
 -10-02 14:52:39.734248 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000008
 -10-02 14:52:39.739306 7fce844ba700 20 proceeding logshard = reshard.0000000009
 -10-02 14:52:39.743348 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000009
 -10-02 14:52:39.748790 7fce844ba700 20 proceeding logshard = reshard.0000000010
 -10-02 14:52:39.750925 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000010
 -10-02 14:52:39.756819 7fce844ba700 20 proceeding logshard = reshard.0000000011
 -10-02 14:52:39.758671 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000011
 -10-02 14:52:39.763882 7fce844ba700 20 proceeding logshard = reshard.0000000012
 -10-02 14:52:39.766274 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000012
 -10-02 14:52:39.774408 7fce844ba700 20 proceeding logshard = reshard.0000000013
 -10-02 14:52:39.777224 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000013
 -10-02 14:52:39.782516 7fce844ba700 20 proceeding logshard = reshard.0000000014
 -10-02 14:52:39.785595 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000014
 -10-02 14:52:39.792351 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000015
 -10-02 14:52:39.798536 7fce844ba700 20 proceeding logshard = reshard.0000000015
 -10-02 14:52:39.801304 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000016
 -10-02 14:52:39.806265 7fce844ba700  2 object expiration: stop
 -10-02 14:52:39.809968 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000017
 -10-02 14:52:39.817940 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000018
 -10-02 14:52:39.825037 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000019
 -10-02 14:52:39.835679 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000020
 -10-02 14:52:39.846239 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000021
 -10-02 14:52:39.854418 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000022
 -10-02 14:52:39.862340 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000023
 -10-02 14:52:39.869550 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000024
 -10-02 14:52:39.876976 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000025
 -10-02 14:52:39.889243 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000026
 -10-02 14:52:39.895881 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000027
 -10-02 14:52:39.902823 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000028
 -10-02 14:52:39.910677 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000029
 -10-02 14:52:39.919699 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000030
 -10-02 14:52:39.927824 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000031
 -10-02 14:52:39.936332 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000032
 -10-02 14:52:39.937822 7fce9c4ea700  2 garbage collection: stop
 -10-02 14:52:39.949177 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000033
 -10-02 14:52:39.958165 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000034
 -10-02 14:52:39.968849 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000035
 -10-02 14:52:39.977742 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000036
 -10-02 14:52:39.985038 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000037
 -10-02 14:52:39.992903 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000038
 -10-02 14:52:40.000812 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000039
 -10-02 14:52:40.009827 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000040
 -10-02 14:52:40.016348 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000041
 -10-02 14:52:40.025094 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000042
 -10-02 14:52:40.034077 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000043
 -10-02 14:52:40.043812 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000044
 -10-02 14:52:40.052986 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000045
 -10-02 14:52:40.062116 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000046
 -10-02 14:52:40.070891 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000047
 -10-02 14:52:40.079881 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000048
 -10-02 14:52:40.088292 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000049
 -10-02 14:52:40.097085 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000050
 -10-02 14:52:40.105351 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000051
 -10-02 14:52:40.113145 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000052
 -10-02 14:52:40.120571 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000053
 -10-02 14:52:40.128560 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000054
 -10-02 14:52:40.136038 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000055
 -10-02 14:52:40.144960 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000056
 -10-02 14:52:40.153432 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000057
 -10-02 14:52:40.162048 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000058
 -10-02 14:52:40.170246 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000059
 -10-02 14:52:40.180787 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000060
 -10-02 14:52:40.190219 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000061
 -10-02 14:52:40.197982 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000062
 -10-02 14:52:40.206306 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000063
 -10-02 14:52:40.213381 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000064
 -10-02 14:52:40.221346 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000065
 -10-02 14:52:40.228516 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000066
 -10-02 14:52:40.237047 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000067
 -10-02 14:52:40.243923 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000068
 -10-02 14:52:40.251114 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000069
 -10-02 14:52:40.259939 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000070
 -10-02 14:52:40.266713 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000071
 -10-02 14:52:40.273895 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000072
 -10-02 14:52:40.281287 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000073
 -10-02 14:52:40.289224 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000074
 -10-02 14:52:40.297111 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000075
 -10-02 14:52:40.305239 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000076
 -10-02 14:52:40.313405 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000077
 -10-02 14:52:40.321270 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000078
 -10-02 14:52:40.328886 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000079
 -10-02 14:52:40.337908 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000080
 -10-02 14:52:40.346531 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000081
 -10-02 14:52:40.355379 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000082
 -10-02 14:52:40.363732 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000083
 -10-02 14:52:40.371801 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000084
 -10-02 14:52:40.384233 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000085
 -10-02 14:52:40.392686 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000086
 -10-02 14:52:40.400489 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000087
 -10-02 14:52:40.409036 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000088
 -10-02 14:52:40.416711 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000089
 -10-02 14:52:40.425332 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000090
 -10-02 14:52:40.433253 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000091
 -10-02 14:52:40.441718 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000092
 -10-02 14:52:40.450422 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000093
 -10-02 14:52:40.458684 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000094
 -10-02 14:52:40.467564 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000095
 -10-02 14:52:40.480059 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000096
 -10-02 14:52:40.487597 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000097
 -10-02 14:52:40.495991 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000098
 -10-02 14:52:40.504346 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000099
 -10-02 14:52:40.513312 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000100
 -10-02 14:52:40.522759 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000101
 -10-02 14:52:40.531681 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000102
 -10-02 14:52:40.543228 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000103
 -10-02 14:52:40.551765 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000104
 -10-02 14:52:40.558688 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000105
 -10-02 14:52:40.566059 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000106
 -10-02 14:52:40.573154 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000107
 -10-02 14:52:40.581706 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000108
 -10-02 14:52:40.589889 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000109
 -10-02 14:52:40.598589 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000110
 -10-02 14:52:40.606265 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000111
 -10-02 14:52:40.620764 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000112
 -10-02 14:52:40.629217 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000113
 -10-02 14:52:40.637280 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000114
 -10-02 14:52:40.645029 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000115
 -10-02 14:52:40.652352 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000116
 -10-02 14:52:40.662665 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000117
 -10-02 14:52:40.677463 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000118
 -10-02 14:52:40.684623 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000119
 -10-02 14:52:40.693025 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000120
 -10-02 14:52:40.701267 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000121
 -10-02 14:52:40.711717 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000122
 -10-02 14:52:40.720431 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000123
 -10-02 14:52:40.728744 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000124
 -10-02 14:52:40.738194 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000125
 -10-02 14:52:40.747427 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000126
 -10-02 14:52:40.756288 7fce9bce9700  2 object expiration: stop
 -10-02 14:53:01.291918 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:53:23.292056 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:53:45.292172 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:54:07.292305 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:54:18.686434 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246408 cookie 217660536320 notifier 266445 bl.length()=695
 -10-02 14:54:18.686526 7fce9dced700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 info.flags=0x17
 -10-02 14:54:18.686540 7fce9dced700 10 adding default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 to cache LRU end
 -10-02 14:54:18.686546 7fce9dced700 10 updating xattr: name=user.rgw.acl bl.length()=163
 -10-02 14:54:18.686548 7fce9dced700 10 updating xattr: name=user.rgw.idtag bl.length()=0
 -10-02 14:54:18.815456 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246409 cookie 217660536320 notifier 266445 bl.length()=764
 -10-02 14:54:18.815492 7fce9dced700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266391.1 info.flags=0x17
 -10-02 14:54:18.815503 7fce9dced700 10 adding default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266391.1 to cache LRU end
 -10-02 14:54:18.815508 7fce9dced700 10 updating xattr: name=user.rgw.acl bl.length()=163
 -10-02 14:54:18.815509 7fce9dced700 10 updating xattr: name=user.rgw.idtag bl.length()=0
 -10-02 14:54:29.292442 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:54:51.292561 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:55:13.292673 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:55:35.292792 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:55:57.292917 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:56:19.293043 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:56:41.293162 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:57:03.293298 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:57:25.293424 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:57:47.293551 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:58:09.293666 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:58:31.293784 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:58:53.293905 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:59:15.294027 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:59:37.294154 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 14:59:59.294274 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:00:21.294418 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:00:43.294542 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:01:05.294679 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:01:27.294801 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:01:49.294907 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:02:11.295038 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:02:33.295162 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:02:39.756409 7fce9bce9700  2 object expiration: start
 -10-02 15:02:39.756430 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000000
 -10-02 15:02:39.764520 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000001
 -10-02 15:02:39.774412 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000002
 -10-02 15:02:39.783328 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000003
 -10-02 15:02:39.791461 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000004
 -10-02 15:02:39.798748 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000005
 -10-02 15:02:39.806291 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000006
 -10-02 15:02:39.806347 7fce844ba700  2 object expiration: start
 -10-02 15:02:39.806358 7fce844ba700 20 proceeding logshard = reshard.0000000000
 -10-02 15:02:39.816079 7fce844ba700 20 proceeding logshard = reshard.0000000001
 -10-02 15:02:39.818164 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000007
 -10-02 15:02:39.822859 7fce844ba700 20 proceeding logshard = reshard.0000000002
 -10-02 15:02:39.827000 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000008
 -10-02 15:02:39.835507 7fce844ba700 20 proceeding logshard = reshard.0000000003
 -10-02 15:02:39.835784 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000009
 -10-02 15:02:39.842896 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000010
 -10-02 15:02:39.845617 7fce844ba700 20 proceeding logshard = reshard.0000000004
 -10-02 15:02:39.850780 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000011
 -10-02 15:02:39.853781 7fce844ba700 20 proceeding logshard = reshard.0000000005
 -10-02 15:02:39.858222 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000012
 -10-02 15:02:39.863511 7fce844ba700 20 proceeding logshard = reshard.0000000006
 -10-02 15:02:39.870440 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000013
 -10-02 15:02:39.872940 7fce844ba700 20 proceeding logshard = reshard.0000000007
 -10-02 15:02:39.879122 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000014
 -10-02 15:02:39.880883 7fce844ba700 20 proceeding logshard = reshard.0000000008
 -10-02 15:02:39.885380 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000015
 -10-02 15:02:39.888748 7fce844ba700 20 proceeding logshard = reshard.0000000009
 -10-02 15:02:39.896303 7fce844ba700 20 proceeding logshard = reshard.0000000010
 -10-02 15:02:39.898670 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000016
 -10-02 15:02:39.905543 7fce844ba700 20 proceeding logshard = reshard.0000000011
 -10-02 15:02:39.906909 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000017
 -10-02 15:02:39.915879 7fce844ba700 20 proceeding logshard = reshard.0000000012
 -10-02 15:02:39.916046 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000018
 -10-02 15:02:39.923652 7fce844ba700 20 proceeding logshard = reshard.0000000013
 -10-02 15:02:39.925722 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000019
 -10-02 15:02:39.931638 7fce844ba700 20 proceeding logshard = reshard.0000000014
 -10-02 15:02:39.932140 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000020
 -10-02 15:02:39.940246 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000021
 -10-02 15:02:39.940626 7fce844ba700 20 proceeding logshard = reshard.0000000015
 -10-02 15:02:39.947949 7fce844ba700  2 object expiration: stop
 -10-02 15:02:39.951058 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000022
 -10-02 15:02:39.959271 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000023
 -10-02 15:02:39.967241 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000024
 -10-02 15:02:39.975455 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000025
 -10-02 15:02:39.984278 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000026
 -10-02 15:02:39.992328 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000027
 -10-02 15:02:39.999937 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000028
 -10-02 15:02:40.034538 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000029
 -10-02 15:02:40.042493 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000030
 -10-02 15:02:40.050255 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000031
 -10-02 15:02:40.059005 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000032
 -10-02 15:02:40.065970 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000033
 -10-02 15:02:40.073446 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000034
 -10-02 15:02:40.082248 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000035
 -10-02 15:02:40.090893 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000036
 -10-02 15:02:40.100179 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000037
 -10-02 15:02:40.108531 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000038
 -10-02 15:02:40.117246 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000039
 -10-02 15:02:40.123850 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000040
 -10-02 15:02:40.131482 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000041
 -10-02 15:02:40.139186 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000042
 -10-02 15:02:40.147111 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000043
 -10-02 15:02:40.155559 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000044
 -10-02 15:02:40.164080 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000045
 -10-02 15:02:40.171653 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000046
 -10-02 15:02:40.183795 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000047
 -10-02 15:02:40.193556 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000048
 -10-02 15:02:40.201827 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000049
 -10-02 15:02:40.209842 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000050
 -10-02 15:02:40.221202 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000051
 -10-02 15:02:40.229971 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000052
 -10-02 15:02:40.237389 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000053
 -10-02 15:02:40.244693 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000054
 -10-02 15:02:40.252030 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000055
 -10-02 15:02:40.259541 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000056
 -10-02 15:02:40.266482 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000057
 -10-02 15:02:40.272126 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000058
 -10-02 15:02:40.278819 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000059
 -10-02 15:02:40.286233 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000060
 -10-02 15:02:40.293359 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000061
 -10-02 15:02:40.300999 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000062
 -10-02 15:02:40.313924 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000063
 -10-02 15:02:40.321276 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000064
 -10-02 15:02:40.329002 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000065
 -10-02 15:02:40.336957 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000066
 -10-02 15:02:40.347802 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000067
 -10-02 15:02:40.356861 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000068
 -10-02 15:02:40.364545 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000069
 -10-02 15:02:40.373646 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000070
 -10-02 15:02:40.380698 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000071
 -10-02 15:02:40.389566 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000072
 -10-02 15:02:40.396041 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000073
 -10-02 15:02:40.403976 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000074
 -10-02 15:02:40.410599 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000075
 -10-02 15:02:40.417743 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000076
 -10-02 15:02:40.424978 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000077
 -10-02 15:02:40.432067 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000078
 -10-02 15:02:40.440238 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000079
 -10-02 15:02:40.447890 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000080
 -10-02 15:02:40.454985 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000081
 -10-02 15:02:40.462615 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000082
 -10-02 15:02:40.470613 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000083
 -10-02 15:02:40.477986 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000084
 -10-02 15:02:40.485846 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000085
 -10-02 15:02:40.494925 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000086
 -10-02 15:02:40.501988 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000087
 -10-02 15:02:40.509930 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000088
 -10-02 15:02:40.517013 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000089
 -10-02 15:02:40.526818 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000090
 -10-02 15:02:40.534622 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000091
 -10-02 15:02:40.543256 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000092
 -10-02 15:02:40.549971 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000093
 -10-02 15:02:40.556459 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000094
 -10-02 15:02:40.565297 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000095
 -10-02 15:02:40.573042 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000096
 -10-02 15:02:40.579414 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000097
 -10-02 15:02:40.586876 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000098
 -10-02 15:02:40.593970 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000099
 -10-02 15:02:40.601411 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000100
 -10-02 15:02:40.609212 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000101
 -10-02 15:02:40.618582 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000102
 -10-02 15:02:40.628529 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000103
 -10-02 15:02:40.636731 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000104
 -10-02 15:02:40.643559 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000105
 -10-02 15:02:40.650913 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000106
 -10-02 15:02:40.659664 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000107
 -10-02 15:02:40.666283 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000108
 -10-02 15:02:40.676757 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000109
 -10-02 15:02:40.685324 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000110
 -10-02 15:02:40.692618 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000111
 -10-02 15:02:40.701160 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000112
 -10-02 15:02:40.707812 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000113
 -10-02 15:02:40.714679 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000114
 -10-02 15:02:40.722360 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000115
 -10-02 15:02:40.729379 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000116
 -10-02 15:02:40.736834 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000117
 -10-02 15:02:40.745126 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000118
 -10-02 15:02:40.753123 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000119
 -10-02 15:02:40.764192 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000120
 -10-02 15:02:40.771209 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000121
 -10-02 15:02:40.778924 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000122
 -10-02 15:02:40.785463 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000123
 -10-02 15:02:40.792048 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000124
 -10-02 15:02:40.800273 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000125
 -10-02 15:02:40.808506 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000126
 -10-02 15:02:40.814620 7fce9bce9700  2 object expiration: stop
 -10-02 15:02:55.295285 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:03:17.295405 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:03:39.295544 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:04:01.295659 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:04:23.295761 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:04:45.295884 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:05:07.296004 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:05:29.296102 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:05:51.296221 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:06:13.296359 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:06:35.296485 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:06:57.296605 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:07:19.296730 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:07:24.375473 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246410 cookie 217660536320 notifier 266445 bl.length()=695
 -10-02 15:07:24.375522 7fce9dced700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 info.flags=0x17
 -10-02 15:07:24.375529 7fce9dced700 10 moving default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 to cache LRU end
 -10-02 15:07:24.375539 7fce9dced700 10 updating xattr: name=user.rgw.acl bl.length()=163
 -10-02 15:07:24.375542 7fce9dced700 10 updating xattr: name=user.rgw.idtag bl.length()=0
 -10-02 15:07:24.381507 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246408 cookie 217660532864 notifier 266445 bl.length()=419
 -10-02 15:07:24.381548 7fce9dced700 10 cache put: name=default.rgw.meta+root+image-net info.flags=0x18
 -10-02 15:07:24.381559 7fce9dced700 10 adding default.rgw.meta+root+image-net to cache LRU end
 -10-02 15:07:24.381561 7fce9dced700 10 appending xattr: name=user.rgw.acl bl.length()=246
 -10-02 15:07:24.388891 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246411 cookie 217660536320 notifier 266445 bl.length()=395
 -10-02 15:07:24.388920 7fce9dced700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 info.flags=0x18
 -10-02 15:07:24.388927 7fce9dced700 10 moving default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 to cache LRU end
 -10-02 15:07:24.388930 7fce9dced700 10 appending xattr: name=user.rgw.acl bl.length()=163
 -10-02 15:07:24.402742 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246409 cookie 217660532864 notifier 266445 bl.length()=619
 -10-02 15:07:24.402779 7fce9dced700 10 cache put: name=default.rgw.meta+root+image-net info.flags=0x17
 -10-02 15:07:24.402784 7fce9dced700 10 moving default.rgw.meta+root+image-net to cache LRU end
 -10-02 15:07:24.402791 7fce9dced700 10 updating xattr: name=user.rgw.acl bl.length()=246
 -10-02 15:07:24.402795 7fce9dced700 10 updating xattr: name=user.rgw.idtag bl.length()=0
 -10-02 15:07:24.409222 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246412 cookie 217660536320 notifier 266445 bl.length()=764
 -10-02 15:07:24.409251 7fce9dced700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266391.1 info.flags=0x17
 -10-02 15:07:24.409256 7fce9dced700 10 moving default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266391.1 to cache LRU end
 -10-02 15:07:24.409261 7fce9dced700 10 updating xattr: name=user.rgw.acl bl.length()=163
 -10-02 15:07:24.409263 7fce9dced700 10 updating xattr: name=user.rgw.idtag bl.length()=0
 -10-02 15:07:39.656088 7fce834b8700  2 keystone revoke thread: start
 -10-02 15:07:39.656115 7fce834b8700  0 ERROR: keystone revocation processing returned error r=-22
 -10-02 15:07:41.296844 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:08:03.296970 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:08:25.297059 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:08:47.297193 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:09:09.297330 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:09:31.297451 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:09:53.297579 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:10:15.297693 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:10:37.297812 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
 -10-02 15:10:59.297944 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start

« Previous
1
2
3
Next »

(1-1/3)

Project

General

Profile

Ceph » rgw

Bug #21619 » luminous-rgw-iccluster023.log