Project

General

Profile

luminous-rgw-iccluster023.log

Radosgw debug log #2 - Yoann Moulin, 10/02/2017 01:32 PM

Download (61.9 KB)

 
1
2017-10-02 06:25:01.408719 7fd9a84ba700  0 ERROR: failed to clone shard, completion_mgr.get_next() returned ret=-125
2
2017-10-02 06:25:01.447135 7fd998293700  1 rgw realm reloader: Store closed
3
2017-10-02 06:25:01.998443 7fd998293700  1 rgw realm reloader: Creating new store
4
2017-10-02 06:25:01.998738 7fd998293700  1 mgrc service_daemon_register rgw.iccluster023 metadata {arch=x86_64,ceph_version=ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable),cpu=Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz,distro=ubuntu,distro_description=Ubuntu 16.04.3 LTS,distro_version=16.04,frontend_config#0=civetweb port=127.0.0.1:8080 num_threads=24,frontend_type#0=civetweb,hostname=iccluster023,kernel_description=#116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017,kernel_version=4.4.0-93-generic,mem_swap_kb=2000056,mem_total_kb=131920952,num_handles=1,os=Linux,pid=2891,zone_id=69d2fd65-fcf9-461b-865f-3dbb053803c4,zone_name=default,zonegroup_id=43d23097-56b9-48a6-ad52-de42341be4bd,zonegroup_name=default}
5
2017-10-02 06:25:01.998790 7fd998293700  1 rgw realm reloader: Finishing initialization of new store
6
2017-10-02 06:25:01.998793 7fd998293700  1 rgw realm reloader:  - REST subsystem init
7
2017-10-02 06:25:01.998823 7fd998293700  1 rgw realm reloader:  - user subsystem init
8
2017-10-02 06:25:01.998831 7fd998293700  1 rgw realm reloader:  - user subsystem init
9
2017-10-02 06:25:01.998838 7fd998293700  1 rgw realm reloader:  - usage subsystem init
10
2017-10-02 06:25:01.998911 7fd998293700  1 rgw realm reloader: Resuming frontends with new realm configuration.
11
2017-10-02 06:36:58.428171 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
12
2017-10-02 06:51:58.428271 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
13
2017-10-02 07:06:58.428362 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
14
2017-10-02 07:21:58.428458 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
15
2017-10-02 07:36:58.428520 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
16
2017-10-02 07:51:58.428652 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
17
2017-10-02 08:06:58.428756 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
18
2017-10-02 08:21:58.428866 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
19
2017-10-02 08:36:58.428973 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
20
2017-10-02 08:51:58.429090 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
21
2017-10-02 09:06:58.429192 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
22
2017-10-02 09:21:58.429301 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
23
2017-10-02 09:36:58.429427 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
24
2017-10-02 09:51:58.429542 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
25
2017-10-02 10:06:58.429656 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
26
2017-10-02 10:21:58.429764 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
27
2017-10-02 10:36:58.429880 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
28
2017-10-02 10:51:58.430007 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
29
2017-10-02 11:06:58.430097 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
30
2017-10-02 11:21:58.430242 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
31
2017-10-02 11:36:58.430393 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
32
2017-10-02 11:51:58.430527 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
33
2017-10-02 12:06:58.430651 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
34
2017-10-02 12:21:58.430768 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
35
2017-10-02 12:36:58.430906 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
36
2017-10-02 12:51:58.431033 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
37
2017-10-02 13:06:58.431125 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
38
2017-10-02 13:21:58.431257 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
39
2017-10-02 13:36:58.431380 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
40
2017-10-02 13:51:58.431483 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
41
2017-10-02 14:06:58.431616 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
42
2017-10-02 14:21:58.431748 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
43
2017-10-02 14:36:58.431900 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
44
2017-10-02 14:51:58.432022 7fd9a5cb5700  0 ERROR: keystone revocation processing returned error r=-22
45
2017-10-02 14:52:39.101815 7fd9a52ad700 -1 received  signal: Terminated from  PID: 1 task name: /sbin/init splash  UID: 0
46
2017-10-02 14:52:39.101834 7fd9a52ad700  1 handle_sigterm
47
2017-10-02 14:52:39.101887 7fd9a52ad700  1 handle_sigterm set alarm for 120
48
2017-10-02 14:52:39.101886 7fd9d68cbe80 -1 shutting down
49
2017-10-02 14:52:39.132678 7fd9a84ba700  0 ERROR: failed to clone shard, completion_mgr.get_next() returned ret=-125
50
2017-10-02 14:52:39.171645 7fd9d68cbe80  1 final shutdown
51
2017-10-02 14:52:39.283328 7fceb68d3e80  0 deferred set uid:gid to 64045:64045 (ceph:ceph)
52
2017-10-02 14:52:39.283374 7fceb68d3e80  0 ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable), process (unknown), pid 33119
53
2017-10-02 14:52:39.291750 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39da0 obj=.rgw.root:default.realm state=0x32ada82620 s->prefetch_data=0
54
2017-10-02 14:52:39.291786 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
55
2017-10-02 14:52:39.294025 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
56
2017-10-02 14:52:39.294044 7fceb68d3e80 20 rados->read ofs=0 len=524288
57
2017-10-02 14:52:39.294611 7fceb68d3e80 20 rados->read r=0 bl.length=46
58
2017-10-02 14:52:39.294664 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39e20 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82620 s->prefetch_data=0
59
2017-10-02 14:52:39.296413 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
60
2017-10-02 14:52:39.296430 7fceb68d3e80 20 rados->read ofs=0 len=524288
61
2017-10-02 14:52:39.297295 7fceb68d3e80 20 rados->read r=0 bl.length=107
62
2017-10-02 14:52:39.297348 7fceb68d3e80 20 realm  default b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
63
2017-10-02 14:52:39.297370 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39c10 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82620 s->prefetch_data=0
64
2017-10-02 14:52:39.298182 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
65
2017-10-02 14:52:39.298199 7fceb68d3e80 20 rados->read ofs=0 len=524288
66
2017-10-02 14:52:39.298978 7fceb68d3e80 20 rados->read r=0 bl.length=107
67
2017-10-02 14:52:39.299018 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39d80 obj=.rgw.root:periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch state=0x32ada82620 s->prefetch_data=0
68
2017-10-02 14:52:39.300542 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
69
2017-10-02 14:52:39.300558 7fceb68d3e80 20 rados->read ofs=0 len=524288
70
2017-10-02 14:52:39.301275 7fceb68d3e80 20 rados->read r=0 bl.length=10
71
2017-10-02 14:52:39.301319 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39da0 obj=.rgw.root:periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 state=0x32ada82620 s->prefetch_data=0
72
2017-10-02 14:52:39.302143 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
73
2017-10-02 14:52:39.302162 7fceb68d3e80 20 rados->read ofs=0 len=524288
74
2017-10-02 14:52:39.302967 7fceb68d3e80 20 rados->read r=0 bl.length=846
75
2017-10-02 14:52:39.303034 7fceb68d3e80 20 current period e7bfcb5a-829b-418f-ae26-d6573a5cc8b9
76
2017-10-02 14:52:39.303051 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c394f0 obj=.rgw.root:converted state=0x32ada82620 s->prefetch_data=0
77
2017-10-02 14:52:39.304666 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38940 obj=.rgw.root:default.realm state=0x32ada82e60 s->prefetch_data=0
78
2017-10-02 14:52:39.305594 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
79
2017-10-02 14:52:39.305613 7fceb68d3e80 20 rados->read ofs=0 len=524288
80
2017-10-02 14:52:39.306417 7fceb68d3e80 20 rados->read r=0 bl.length=46
81
2017-10-02 14:52:39.306463 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c389c0 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82e60 s->prefetch_data=0
82
2017-10-02 14:52:39.307343 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
83
2017-10-02 14:52:39.307361 7fceb68d3e80 20 rados->read ofs=0 len=524288
84
2017-10-02 14:52:39.308133 7fceb68d3e80 20 rados->read r=0 bl.length=107
85
2017-10-02 14:52:39.308161 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38ad0 obj=.rgw.root:default.region state=0x32ada82e60 s->prefetch_data=0
86
2017-10-02 14:52:39.557156 7fceb68d3e80 20 RGWRados::pool_iterate: got zonegroup_info.43d23097-56b9-48a6-ad52-de42341be4bd
87
2017-10-02 14:52:39.557176 7fceb68d3e80 20 RGWRados::pool_iterate: got periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch
88
2017-10-02 14:52:39.557180 7fceb68d3e80 20 RGWRados::pool_iterate: got periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1
89
2017-10-02 14:52:39.557182 7fceb68d3e80 20 RGWRados::pool_iterate: got realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
90
2017-10-02 14:52:39.557186 7fceb68d3e80 20 RGWRados::pool_iterate: got default.realm
91
2017-10-02 14:52:39.557188 7fceb68d3e80 20 RGWRados::pool_iterate: got default.zone.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
92
2017-10-02 14:52:39.557210 7fceb68d3e80 20 RGWRados::pool_iterate: got periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch
93
2017-10-02 14:52:39.557213 7fceb68d3e80 20 RGWRados::pool_iterate: got default.zonegroup.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
94
2017-10-02 14:52:39.557215 7fceb68d3e80 20 RGWRados::pool_iterate: got periods.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a:staging
95
2017-10-02 14:52:39.557216 7fceb68d3e80 20 RGWRados::pool_iterate: got realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a.control
96
2017-10-02 14:52:39.557218 7fceb68d3e80 20 RGWRados::pool_iterate: got period_config.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a
97
2017-10-02 14:52:39.557219 7fceb68d3e80 20 RGWRados::pool_iterate: got zone_info.69d2fd65-fcf9-461b-865f-3dbb053803c4
98
2017-10-02 14:52:39.557241 7fceb68d3e80 20 RGWRados::pool_iterate: got realms_names.default
99
2017-10-02 14:52:39.557244 7fceb68d3e80 20 RGWRados::pool_iterate: got zone_names.default
100
2017-10-02 14:52:39.557246 7fceb68d3e80 20 RGWRados::pool_iterate: got zonegroups_names.default
101
2017-10-02 14:52:39.557247 7fceb68d3e80 20 RGWRados::pool_iterate: got periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1
102
2017-10-02 14:52:39.557268 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38bc0 obj=.rgw.root:zone_names.default state=0x32ada82e60 s->prefetch_data=0
103
2017-10-02 14:52:39.558064 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
104
2017-10-02 14:52:39.558083 7fceb68d3e80 20 rados->read ofs=0 len=524288
105
2017-10-02 14:52:39.558704 7fceb68d3e80 20 rados->read r=0 bl.length=46
106
2017-10-02 14:52:39.558764 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38bc0 obj=.rgw.root:zone_info.69d2fd65-fcf9-461b-865f-3dbb053803c4 state=0x32ada82e60 s->prefetch_data=0
107
2017-10-02 14:52:39.560437 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
108
2017-10-02 14:52:39.560455 7fceb68d3e80 20 rados->read ofs=0 len=524288
109
2017-10-02 14:52:39.561322 7fceb68d3e80 20 rados->read r=0 bl.length=724
110
2017-10-02 14:52:39.561375 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38bc0 obj=.rgw.root:zonegroups_names.default state=0x32ada82e60 s->prefetch_data=0
111
2017-10-02 14:52:39.562122 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
112
2017-10-02 14:52:39.562140 7fceb68d3e80 20 rados->read ofs=0 len=524288
113
2017-10-02 14:52:39.562761 7fceb68d3e80 20 rados->read r=0 bl.length=46
114
2017-10-02 14:52:39.562797 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38bc0 obj=.rgw.root:zonegroup_info.43d23097-56b9-48a6-ad52-de42341be4bd state=0x32ada82e60 s->prefetch_data=0
115
2017-10-02 14:52:39.563629 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
116
2017-10-02 14:52:39.563645 7fceb68d3e80 20 rados->read ofs=0 len=524288
117
2017-10-02 14:52:39.564418 7fceb68d3e80 20 rados->read r=0 bl.length=369
118
2017-10-02 14:52:39.564467 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39fb0 obj=.rgw.root:region_map state=0x32ada82620 s->prefetch_data=0
119
2017-10-02 14:52:39.565318 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39630 obj=.rgw.root:default.realm state=0x32ada82620 s->prefetch_data=0
120
2017-10-02 14:52:39.566464 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
121
2017-10-02 14:52:39.566481 7fceb68d3e80 20 rados->read ofs=0 len=524288
122
2017-10-02 14:52:39.567298 7fceb68d3e80 20 rados->read r=0 bl.length=46
123
2017-10-02 14:52:39.567335 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c396b0 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82620 s->prefetch_data=0
124
2017-10-02 14:52:39.568256 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
125
2017-10-02 14:52:39.568274 7fceb68d3e80 20 rados->read ofs=0 len=524288
126
2017-10-02 14:52:39.569137 7fceb68d3e80 20 rados->read r=0 bl.length=107
127
2017-10-02 14:52:39.569184 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c397c0 obj=.rgw.root:default.zonegroup.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82620 s->prefetch_data=0
128
2017-10-02 14:52:39.570143 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
129
2017-10-02 14:52:39.570161 7fceb68d3e80 20 rados->read ofs=0 len=524288
130
2017-10-02 14:52:39.571006 7fceb68d3e80 20 rados->read r=0 bl.length=46
131
2017-10-02 14:52:39.571047 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39a40 obj=.rgw.root:zonegroup_info.43d23097-56b9-48a6-ad52-de42341be4bd state=0x32ada82620 s->prefetch_data=0
132
2017-10-02 14:52:39.571888 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
133
2017-10-02 14:52:39.571906 7fceb68d3e80 20 rados->read ofs=0 len=524288
134
2017-10-02 14:52:39.572636 7fceb68d3e80 20 rados->read r=0 bl.length=369
135
2017-10-02 14:52:39.572675 7fceb68d3e80 20 period zonegroup init ret 0
136
2017-10-02 14:52:39.572678 7fceb68d3e80 20 period zonegroup name default
137
2017-10-02 14:52:39.572681 7fceb68d3e80 20 using current period zonegroup default
138
2017-10-02 14:52:39.572692 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39630 obj=.rgw.root:default.realm state=0x32ada82620 s->prefetch_data=0
139
2017-10-02 14:52:39.576128 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
140
2017-10-02 14:52:39.576144 7fceb68d3e80 20 rados->read ofs=0 len=524288
141
2017-10-02 14:52:39.577040 7fceb68d3e80 20 rados->read r=0 bl.length=46
142
2017-10-02 14:52:39.577077 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c396b0 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82620 s->prefetch_data=0
143
2017-10-02 14:52:39.577993 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
144
2017-10-02 14:52:39.578012 7fceb68d3e80 20 rados->read ofs=0 len=524288
145
2017-10-02 14:52:39.578807 7fceb68d3e80 20 rados->read r=0 bl.length=107
146
2017-10-02 14:52:39.578847 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c397c0 obj=.rgw.root:default.zone.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada82620 s->prefetch_data=0
147
2017-10-02 14:52:39.579606 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
148
2017-10-02 14:52:39.579623 7fceb68d3e80 20 rados->read ofs=0 len=524288
149
2017-10-02 14:52:39.580350 7fceb68d3e80 20 rados->read r=0 bl.length=46
150
2017-10-02 14:52:39.580385 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39a40 obj=.rgw.root:zone_info.69d2fd65-fcf9-461b-865f-3dbb053803c4 state=0x32ada82620 s->prefetch_data=0
151
2017-10-02 14:52:39.581242 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
152
2017-10-02 14:52:39.581259 7fceb68d3e80 20 rados->read ofs=0 len=524288
153
2017-10-02 14:52:39.582137 7fceb68d3e80 20 rados->read r=0 bl.length=724
154
2017-10-02 14:52:39.582190 7fceb68d3e80 10 Cannot find current period zone using local zone
155
2017-10-02 14:52:39.582202 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39e20 obj=.rgw.root:zone_info.69d2fd65-fcf9-461b-865f-3dbb053803c4 state=0x32ada82620 s->prefetch_data=0
156
2017-10-02 14:52:39.583006 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
157
2017-10-02 14:52:39.583023 7fceb68d3e80 20 rados->read ofs=0 len=524288
158
2017-10-02 14:52:39.583809 7fceb68d3e80 20 rados->read r=0 bl.length=724
159
2017-10-02 14:52:39.583870 7fceb68d3e80 20 zone default
160
2017-10-02 14:52:39.592370 7fceb68d3e80 20 add_watcher() i=0
161
2017-10-02 14:52:39.599381 7fceb68d3e80 20 add_watcher() i=1
162
2017-10-02 14:52:39.607364 7fceb68d3e80 20 add_watcher() i=2
163
2017-10-02 14:52:39.614888 7fceb68d3e80 20 add_watcher() i=3
164
2017-10-02 14:52:39.622705 7fceb68d3e80 20 add_watcher() i=4
165
2017-10-02 14:52:39.630970 7fceb68d3e80 20 add_watcher() i=5
166
2017-10-02 14:52:39.639366 7fceb68d3e80 20 add_watcher() i=6
167
2017-10-02 14:52:39.646629 7fceb68d3e80 20 add_watcher() i=7
168
2017-10-02 14:52:39.646646 7fceb68d3e80  2 all 8 watchers are set, enabling cache
169
2017-10-02 14:52:39.646796 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39730 obj=default.rgw.log:meta.history state=0x32ada82620 s->prefetch_data=0
170
2017-10-02 14:52:39.646798 7fce9c4ea700  2 garbage collection: start
171
2017-10-02 14:52:39.646809 7fceb68d3e80 10 cache get: name=default.rgw.log++meta.history : miss
172
2017-10-02 14:52:39.647026 7fce9bce9700  2 object expiration: start
173
2017-10-02 14:52:39.647066 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000000
174
2017-10-02 14:52:39.647679 7fceb68d3e80 10 cache put: name=default.rgw.log++meta.history info.flags=0x16
175
2017-10-02 14:52:39.647726 7fceb68d3e80 10 adding default.rgw.log++meta.history to cache LRU end
176
2017-10-02 14:52:39.647735 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
177
2017-10-02 14:52:39.647742 7fceb68d3e80 10 cache get: name=default.rgw.log++meta.history : type miss (requested=0x11, cached=0x16)
178
2017-10-02 14:52:39.647762 7fceb68d3e80 20 rados->read ofs=0 len=524288
179
2017-10-02 14:52:39.648802 7fceb68d3e80 20 rados->read r=0 bl.length=50
180
2017-10-02 14:52:39.648825 7fceb68d3e80 10 cache put: name=default.rgw.log++meta.history info.flags=0x11
181
2017-10-02 14:52:39.648829 7fceb68d3e80 10 moving default.rgw.log++meta.history to cache LRU end
182
2017-10-02 14:52:39.648867 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38f50 obj=.rgw.root:periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch state=0x32ada82620 s->prefetch_data=0
183
2017-10-02 14:52:39.648880 7fceb68d3e80 10 cache get: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch : miss
184
2017-10-02 14:52:39.649777 7fceb68d3e80 10 cache put: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch info.flags=0x6
185
2017-10-02 14:52:39.649797 7fceb68d3e80 10 adding .rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch to cache LRU end
186
2017-10-02 14:52:39.649803 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
187
2017-10-02 14:52:39.649807 7fceb68d3e80 10 cache get: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch : type miss (requested=0x1, cached=0x6)
188
2017-10-02 14:52:39.649810 7fceb68d3e80 20 rados->read ofs=0 len=524288
189
2017-10-02 14:52:39.650596 7fceb68d3e80 20 rados->read r=0 bl.length=10
190
2017-10-02 14:52:39.650621 7fceb68d3e80 10 cache put: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch info.flags=0x1
191
2017-10-02 14:52:39.650625 7fceb68d3e80 10 moving .rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.latest_epoch to cache LRU end
192
2017-10-02 14:52:39.650648 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c38f70 obj=.rgw.root:periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 state=0x32ada82620 s->prefetch_data=0
193
2017-10-02 14:52:39.650658 7fceb68d3e80 10 cache get: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 : miss
194
2017-10-02 14:52:39.651586 7fceb68d3e80 10 cache put: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 info.flags=0x6
195
2017-10-02 14:52:39.651607 7fceb68d3e80 10 adding .rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 to cache LRU end
196
2017-10-02 14:52:39.651614 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
197
2017-10-02 14:52:39.651620 7fceb68d3e80 10 cache get: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 : type miss (requested=0x1, cached=0x6)
198
2017-10-02 14:52:39.651623 7fceb68d3e80 20 rados->read ofs=0 len=524288
199
2017-10-02 14:52:39.652764 7fceb68d3e80 20 rados->read r=0 bl.length=249
200
2017-10-02 14:52:39.653017 7fceb68d3e80 10 cache put: name=.rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 info.flags=0x1
201
2017-10-02 14:52:39.653028 7fceb68d3e80 10 moving .rgw.root++periods.6af5cc56-b57a-4433-a8fd-ab4460098092.1 to cache LRU end
202
2017-10-02 14:52:39.653048 7fceb68d3e80 14 rgw period puller: found period 6af5cc56-b57a-4433-a8fd-ab4460098092 in local storage
203
2017-10-02 14:52:39.654517 7fce8acc7700 20 reqs_thread_entry: start
204
2017-10-02 14:52:39.655007 7fce884c2700 20 reqs_thread_entry: start
205
2017-10-02 14:52:39.655201 7fce87cc1700 20 cr:s=0x32add86000:op=0x32adab0800:20MetaMasterTrimPollCR: operate()
206
2017-10-02 14:52:39.655246 7fce87cc1700 20 run: stack=0x32add86000 is io blocked
207
2017-10-02 14:52:39.655255 7fce87cc1700 20 cr:s=0x32add860f0:op=0x32adb46700:17DataLogTrimPollCR: operate()
208
2017-10-02 14:52:39.655274 7fce87cc1700 20 run: stack=0x32add860f0 is io blocked
209
2017-10-02 14:52:39.655322 7fce86cbf700 20 reqs_thread_entry: start
210
2017-10-02 14:52:39.655383 7fce854bc700 20 BucketsSyncThread: start
211
2017-10-02 14:52:39.655433 7fceb68d3e80 20 init_complete bucket index max shards: 0
212
2017-10-02 14:52:39.655447 7fce84cbb700 20 UserSyncThread: start
213
2017-10-02 14:52:39.655431 7fce85cbd700  5 schedule life cycle next start time: Mon Oct  2 22:00:00 2017
214
2017-10-02 14:52:39.655470 7fce84cbb700 10 ERROR: can't get key: ret=-2
215
2017-10-02 14:52:39.655477 7fce84cbb700  5 ERROR: sync_all_users() returned ret=-2
216
2017-10-02 14:52:39.655500 7fce844ba700  2 object expiration: start
217
2017-10-02 14:52:39.655515 7fce844ba700 20 proceeding logshard = reshard.0000000000
218
2017-10-02 14:52:39.655704 7fceb68d3e80 20 RGW hostnames: test.iccluster.epfl.ch
219
2017-10-02 14:52:39.655712 7fceb68d3e80 20 RGW S3website hostnames: 
220
2017-10-02 14:52:39.655945 7fce834b8700  2 keystone revoke thread: start
221
2017-10-02 14:52:39.655957 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000001
222
2017-10-02 14:52:39.655978 7fce834b8700  0 ERROR: keystone revocation processing returned error r=-22
223
2017-10-02 14:52:39.657718 7fceb68d3e80  0 starting handler: civetweb
224
2017-10-02 14:52:39.657744 7fceb68d3e80 20 civetweb config: canonicalize_url_path: no
225
2017-10-02 14:52:39.657747 7fceb68d3e80 20 civetweb config: decode_url: no
226
2017-10-02 14:52:39.657749 7fceb68d3e80 20 civetweb config: enable_auth_domain_check: no
227
2017-10-02 14:52:39.657750 7fceb68d3e80 20 civetweb config: enable_keep_alive: yes
228
2017-10-02 14:52:39.657753 7fceb68d3e80 20 civetweb config: listening_ports: 127.0.0.1:8080
229
2017-10-02 14:52:39.657755 7fceb68d3e80 20 civetweb config: num_threads: 24
230
2017-10-02 14:52:39.657757 7fceb68d3e80 20 civetweb config: run_as_user: ceph
231
2017-10-02 14:52:39.657758 7fceb68d3e80 20 civetweb config: validate_http_method: no
232
2017-10-02 14:52:39.660368 7fceb68d3e80  1 mgrc service_daemon_register rgw.iccluster023 metadata {arch=x86_64,ceph_version=ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable),cpu=Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz,distro=ubuntu,distro_description=Ubuntu 16.04.3 LTS,distro_version=16.04,frontend_config#0=civetweb port=127.0.0.1:8080 num_threads=24,frontend_type#0=civetweb,hostname=iccluster023,kernel_description=#116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017,kernel_version=4.4.0-93-generic,mem_swap_kb=2000056,mem_total_kb=131920952,num_handles=1,os=Linux,pid=33119,zone_id=69d2fd65-fcf9-461b-865f-3dbb053803c4,zone_name=default,zonegroup_id=43d23097-56b9-48a6-ad52-de42341be4bd,zonegroup_name=default}
233
2017-10-02 14:52:39.660435 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39c70 obj=.rgw.root:realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a state=0x32ada83120 s->prefetch_data=0
234
2017-10-02 14:52:39.660446 7fceb68d3e80 10 cache get: name=.rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a : miss
235
2017-10-02 14:52:39.661236 7fceb68d3e80 10 cache put: name=.rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a info.flags=0x6
236
2017-10-02 14:52:39.661252 7fceb68d3e80 10 adding .rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a to cache LRU end
237
2017-10-02 14:52:39.661257 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
238
2017-10-02 14:52:39.661262 7fceb68d3e80 10 cache get: name=.rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a : type miss (requested=0x1, cached=0x6)
239
2017-10-02 14:52:39.661265 7fceb68d3e80 20 rados->read ofs=0 len=524288
240
2017-10-02 14:52:39.662032 7fceb68d3e80 20 rados->read r=0 bl.length=107
241
2017-10-02 14:52:39.662045 7fceb68d3e80 10 cache put: name=.rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a info.flags=0x1
242
2017-10-02 14:52:39.662050 7fceb68d3e80 10 moving .rgw.root++realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a to cache LRU end
243
2017-10-02 14:52:39.662069 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39de0 obj=.rgw.root:periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch state=0x32ada83120 s->prefetch_data=0
244
2017-10-02 14:52:39.662086 7fceb68d3e80 10 cache get: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch : miss
245
2017-10-02 14:52:39.662727 7fceb68d3e80 10 cache put: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch info.flags=0x6
246
2017-10-02 14:52:39.662740 7fceb68d3e80 10 adding .rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch to cache LRU end
247
2017-10-02 14:52:39.662744 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
248
2017-10-02 14:52:39.662747 7fceb68d3e80 10 cache get: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch : type miss (requested=0x1, cached=0x6)
249
2017-10-02 14:52:39.662749 7fceb68d3e80 20 rados->read ofs=0 len=524288
250
2017-10-02 14:52:39.663236 7fceb68d3e80 20 rados->read r=0 bl.length=10
251
2017-10-02 14:52:39.663253 7fceb68d3e80 10 cache put: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch info.flags=0x1
252
2017-10-02 14:52:39.663256 7fceb68d3e80 10 moving .rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.latest_epoch to cache LRU end
253
2017-10-02 14:52:39.663295 7fceb68d3e80 20 get_system_obj_state: rctx=0x7ffcf6c39e00 obj=.rgw.root:periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 state=0x32ada83120 s->prefetch_data=0
254
2017-10-02 14:52:39.663304 7fceb68d3e80 10 cache get: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 : miss
255
2017-10-02 14:52:39.664021 7fceb68d3e80 10 cache put: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 info.flags=0x6
256
2017-10-02 14:52:39.664033 7fceb68d3e80 10 adding .rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 to cache LRU end
257
2017-10-02 14:52:39.664037 7fceb68d3e80 20 get_system_obj_state: s->obj_tag was set empty
258
2017-10-02 14:52:39.664041 7fceb68d3e80 10 cache get: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 : type miss (requested=0x1, cached=0x6)
259
2017-10-02 14:52:39.664043 7fceb68d3e80 20 rados->read ofs=0 len=524288
260
2017-10-02 14:52:39.664742 7fceb68d3e80 20 rados->read r=0 bl.length=846
261
2017-10-02 14:52:39.664756 7fceb68d3e80 10 cache put: name=.rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 info.flags=0x1
262
2017-10-02 14:52:39.664759 7fceb68d3e80 10 moving .rgw.root++periods.e7bfcb5a-829b-418f-ae26-d6573a5cc8b9.1 to cache LRU end
263
2017-10-02 14:52:39.664790 7fceb68d3e80  4 rgw period pusher: No zones to update
264
2017-10-02 14:52:39.667182 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000002
265
2017-10-02 14:52:39.667365 7fce844ba700 20 proceeding logshard = reshard.0000000001
266
2017-10-02 14:52:39.672326 7fceb68d3e80 10 rgw realm watcher: Watching realms.b5cc8a8e-bd96-4b19-8cdd-e87a58ed518a.control
267
2017-10-02 14:52:39.677440 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000003
268
2017-10-02 14:52:39.677801 7fce844ba700 20 proceeding logshard = reshard.0000000002
269
2017-10-02 14:52:39.686557 7fce844ba700 20 proceeding logshard = reshard.0000000003
270
2017-10-02 14:52:39.687589 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000004
271
2017-10-02 14:52:39.694650 7fce844ba700 20 proceeding logshard = reshard.0000000004
272
2017-10-02 14:52:39.704502 7fce844ba700 20 proceeding logshard = reshard.0000000005
273
2017-10-02 14:52:39.706923 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000005
274
2017-10-02 14:52:39.711889 7fce844ba700 20 proceeding logshard = reshard.0000000006
275
2017-10-02 14:52:39.715845 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000006
276
2017-10-02 14:52:39.721913 7fce844ba700 20 proceeding logshard = reshard.0000000007
277
2017-10-02 14:52:39.725995 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000007
278
2017-10-02 14:52:39.730904 7fce844ba700 20 proceeding logshard = reshard.0000000008
279
2017-10-02 14:52:39.734248 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000008
280
2017-10-02 14:52:39.739306 7fce844ba700 20 proceeding logshard = reshard.0000000009
281
2017-10-02 14:52:39.743348 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000009
282
2017-10-02 14:52:39.748790 7fce844ba700 20 proceeding logshard = reshard.0000000010
283
2017-10-02 14:52:39.750925 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000010
284
2017-10-02 14:52:39.756819 7fce844ba700 20 proceeding logshard = reshard.0000000011
285
2017-10-02 14:52:39.758671 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000011
286
2017-10-02 14:52:39.763882 7fce844ba700 20 proceeding logshard = reshard.0000000012
287
2017-10-02 14:52:39.766274 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000012
288
2017-10-02 14:52:39.774408 7fce844ba700 20 proceeding logshard = reshard.0000000013
289
2017-10-02 14:52:39.777224 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000013
290
2017-10-02 14:52:39.782516 7fce844ba700 20 proceeding logshard = reshard.0000000014
291
2017-10-02 14:52:39.785595 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000014
292
2017-10-02 14:52:39.792351 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000015
293
2017-10-02 14:52:39.798536 7fce844ba700 20 proceeding logshard = reshard.0000000015
294
2017-10-02 14:52:39.801304 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000016
295
2017-10-02 14:52:39.806265 7fce844ba700  2 object expiration: stop
296
2017-10-02 14:52:39.809968 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000017
297
2017-10-02 14:52:39.817940 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000018
298
2017-10-02 14:52:39.825037 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000019
299
2017-10-02 14:52:39.835679 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000020
300
2017-10-02 14:52:39.846239 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000021
301
2017-10-02 14:52:39.854418 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000022
302
2017-10-02 14:52:39.862340 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000023
303
2017-10-02 14:52:39.869550 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000024
304
2017-10-02 14:52:39.876976 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000025
305
2017-10-02 14:52:39.889243 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000026
306
2017-10-02 14:52:39.895881 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000027
307
2017-10-02 14:52:39.902823 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000028
308
2017-10-02 14:52:39.910677 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000029
309
2017-10-02 14:52:39.919699 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000030
310
2017-10-02 14:52:39.927824 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000031
311
2017-10-02 14:52:39.936332 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000032
312
2017-10-02 14:52:39.937822 7fce9c4ea700  2 garbage collection: stop
313
2017-10-02 14:52:39.949177 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000033
314
2017-10-02 14:52:39.958165 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000034
315
2017-10-02 14:52:39.968849 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000035
316
2017-10-02 14:52:39.977742 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000036
317
2017-10-02 14:52:39.985038 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000037
318
2017-10-02 14:52:39.992903 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000038
319
2017-10-02 14:52:40.000812 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000039
320
2017-10-02 14:52:40.009827 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000040
321
2017-10-02 14:52:40.016348 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000041
322
2017-10-02 14:52:40.025094 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000042
323
2017-10-02 14:52:40.034077 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000043
324
2017-10-02 14:52:40.043812 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000044
325
2017-10-02 14:52:40.052986 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000045
326
2017-10-02 14:52:40.062116 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000046
327
2017-10-02 14:52:40.070891 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000047
328
2017-10-02 14:52:40.079881 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000048
329
2017-10-02 14:52:40.088292 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000049
330
2017-10-02 14:52:40.097085 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000050
331
2017-10-02 14:52:40.105351 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000051
332
2017-10-02 14:52:40.113145 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000052
333
2017-10-02 14:52:40.120571 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000053
334
2017-10-02 14:52:40.128560 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000054
335
2017-10-02 14:52:40.136038 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000055
336
2017-10-02 14:52:40.144960 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000056
337
2017-10-02 14:52:40.153432 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000057
338
2017-10-02 14:52:40.162048 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000058
339
2017-10-02 14:52:40.170246 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000059
340
2017-10-02 14:52:40.180787 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000060
341
2017-10-02 14:52:40.190219 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000061
342
2017-10-02 14:52:40.197982 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000062
343
2017-10-02 14:52:40.206306 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000063
344
2017-10-02 14:52:40.213381 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000064
345
2017-10-02 14:52:40.221346 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000065
346
2017-10-02 14:52:40.228516 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000066
347
2017-10-02 14:52:40.237047 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000067
348
2017-10-02 14:52:40.243923 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000068
349
2017-10-02 14:52:40.251114 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000069
350
2017-10-02 14:52:40.259939 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000070
351
2017-10-02 14:52:40.266713 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000071
352
2017-10-02 14:52:40.273895 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000072
353
2017-10-02 14:52:40.281287 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000073
354
2017-10-02 14:52:40.289224 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000074
355
2017-10-02 14:52:40.297111 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000075
356
2017-10-02 14:52:40.305239 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000076
357
2017-10-02 14:52:40.313405 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000077
358
2017-10-02 14:52:40.321270 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000078
359
2017-10-02 14:52:40.328886 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000079
360
2017-10-02 14:52:40.337908 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000080
361
2017-10-02 14:52:40.346531 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000081
362
2017-10-02 14:52:40.355379 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000082
363
2017-10-02 14:52:40.363732 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000083
364
2017-10-02 14:52:40.371801 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000084
365
2017-10-02 14:52:40.384233 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000085
366
2017-10-02 14:52:40.392686 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000086
367
2017-10-02 14:52:40.400489 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000087
368
2017-10-02 14:52:40.409036 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000088
369
2017-10-02 14:52:40.416711 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000089
370
2017-10-02 14:52:40.425332 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000090
371
2017-10-02 14:52:40.433253 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000091
372
2017-10-02 14:52:40.441718 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000092
373
2017-10-02 14:52:40.450422 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000093
374
2017-10-02 14:52:40.458684 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000094
375
2017-10-02 14:52:40.467564 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000095
376
2017-10-02 14:52:40.480059 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000096
377
2017-10-02 14:52:40.487597 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000097
378
2017-10-02 14:52:40.495991 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000098
379
2017-10-02 14:52:40.504346 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000099
380
2017-10-02 14:52:40.513312 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000100
381
2017-10-02 14:52:40.522759 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000101
382
2017-10-02 14:52:40.531681 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000102
383
2017-10-02 14:52:40.543228 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000103
384
2017-10-02 14:52:40.551765 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000104
385
2017-10-02 14:52:40.558688 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000105
386
2017-10-02 14:52:40.566059 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000106
387
2017-10-02 14:52:40.573154 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000107
388
2017-10-02 14:52:40.581706 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000108
389
2017-10-02 14:52:40.589889 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000109
390
2017-10-02 14:52:40.598589 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000110
391
2017-10-02 14:52:40.606265 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000111
392
2017-10-02 14:52:40.620764 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000112
393
2017-10-02 14:52:40.629217 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000113
394
2017-10-02 14:52:40.637280 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000114
395
2017-10-02 14:52:40.645029 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000115
396
2017-10-02 14:52:40.652352 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000116
397
2017-10-02 14:52:40.662665 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000117
398
2017-10-02 14:52:40.677463 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000118
399
2017-10-02 14:52:40.684623 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000119
400
2017-10-02 14:52:40.693025 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000120
401
2017-10-02 14:52:40.701267 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000121
402
2017-10-02 14:52:40.711717 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000122
403
2017-10-02 14:52:40.720431 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000123
404
2017-10-02 14:52:40.728744 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000124
405
2017-10-02 14:52:40.738194 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000125
406
2017-10-02 14:52:40.747427 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000126
407
2017-10-02 14:52:40.756288 7fce9bce9700  2 object expiration: stop
408
2017-10-02 14:53:01.291918 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
409
2017-10-02 14:53:23.292056 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
410
2017-10-02 14:53:45.292172 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
411
2017-10-02 14:54:07.292305 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
412
2017-10-02 14:54:18.686434 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246408 cookie 217660536320 notifier 266445 bl.length()=695
413
2017-10-02 14:54:18.686526 7fce9dced700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 info.flags=0x17
414
2017-10-02 14:54:18.686540 7fce9dced700 10 adding default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 to cache LRU end
415
2017-10-02 14:54:18.686546 7fce9dced700 10 updating xattr: name=user.rgw.acl bl.length()=163
416
2017-10-02 14:54:18.686548 7fce9dced700 10 updating xattr: name=user.rgw.idtag bl.length()=0
417
2017-10-02 14:54:18.815456 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246409 cookie 217660536320 notifier 266445 bl.length()=764
418
2017-10-02 14:54:18.815492 7fce9dced700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266391.1 info.flags=0x17
419
2017-10-02 14:54:18.815503 7fce9dced700 10 adding default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266391.1 to cache LRU end
420
2017-10-02 14:54:18.815508 7fce9dced700 10 updating xattr: name=user.rgw.acl bl.length()=163
421
2017-10-02 14:54:18.815509 7fce9dced700 10 updating xattr: name=user.rgw.idtag bl.length()=0
422
2017-10-02 14:54:29.292442 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
423
2017-10-02 14:54:51.292561 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
424
2017-10-02 14:55:13.292673 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
425
2017-10-02 14:55:35.292792 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
426
2017-10-02 14:55:57.292917 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
427
2017-10-02 14:56:19.293043 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
428
2017-10-02 14:56:41.293162 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
429
2017-10-02 14:57:03.293298 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
430
2017-10-02 14:57:25.293424 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
431
2017-10-02 14:57:47.293551 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
432
2017-10-02 14:58:09.293666 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
433
2017-10-02 14:58:31.293784 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
434
2017-10-02 14:58:53.293905 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
435
2017-10-02 14:59:15.294027 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
436
2017-10-02 14:59:37.294154 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
437
2017-10-02 14:59:59.294274 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
438
2017-10-02 15:00:21.294418 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
439
2017-10-02 15:00:43.294542 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
440
2017-10-02 15:01:05.294679 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
441
2017-10-02 15:01:27.294801 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
442
2017-10-02 15:01:49.294907 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
443
2017-10-02 15:02:11.295038 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
444
2017-10-02 15:02:33.295162 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
445
2017-10-02 15:02:39.756409 7fce9bce9700  2 object expiration: start
446
2017-10-02 15:02:39.756430 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000000
447
2017-10-02 15:02:39.764520 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000001
448
2017-10-02 15:02:39.774412 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000002
449
2017-10-02 15:02:39.783328 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000003
450
2017-10-02 15:02:39.791461 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000004
451
2017-10-02 15:02:39.798748 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000005
452
2017-10-02 15:02:39.806291 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000006
453
2017-10-02 15:02:39.806347 7fce844ba700  2 object expiration: start
454
2017-10-02 15:02:39.806358 7fce844ba700 20 proceeding logshard = reshard.0000000000
455
2017-10-02 15:02:39.816079 7fce844ba700 20 proceeding logshard = reshard.0000000001
456
2017-10-02 15:02:39.818164 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000007
457
2017-10-02 15:02:39.822859 7fce844ba700 20 proceeding logshard = reshard.0000000002
458
2017-10-02 15:02:39.827000 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000008
459
2017-10-02 15:02:39.835507 7fce844ba700 20 proceeding logshard = reshard.0000000003
460
2017-10-02 15:02:39.835784 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000009
461
2017-10-02 15:02:39.842896 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000010
462
2017-10-02 15:02:39.845617 7fce844ba700 20 proceeding logshard = reshard.0000000004
463
2017-10-02 15:02:39.850780 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000011
464
2017-10-02 15:02:39.853781 7fce844ba700 20 proceeding logshard = reshard.0000000005
465
2017-10-02 15:02:39.858222 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000012
466
2017-10-02 15:02:39.863511 7fce844ba700 20 proceeding logshard = reshard.0000000006
467
2017-10-02 15:02:39.870440 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000013
468
2017-10-02 15:02:39.872940 7fce844ba700 20 proceeding logshard = reshard.0000000007
469
2017-10-02 15:02:39.879122 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000014
470
2017-10-02 15:02:39.880883 7fce844ba700 20 proceeding logshard = reshard.0000000008
471
2017-10-02 15:02:39.885380 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000015
472
2017-10-02 15:02:39.888748 7fce844ba700 20 proceeding logshard = reshard.0000000009
473
2017-10-02 15:02:39.896303 7fce844ba700 20 proceeding logshard = reshard.0000000010
474
2017-10-02 15:02:39.898670 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000016
475
2017-10-02 15:02:39.905543 7fce844ba700 20 proceeding logshard = reshard.0000000011
476
2017-10-02 15:02:39.906909 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000017
477
2017-10-02 15:02:39.915879 7fce844ba700 20 proceeding logshard = reshard.0000000012
478
2017-10-02 15:02:39.916046 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000018
479
2017-10-02 15:02:39.923652 7fce844ba700 20 proceeding logshard = reshard.0000000013
480
2017-10-02 15:02:39.925722 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000019
481
2017-10-02 15:02:39.931638 7fce844ba700 20 proceeding logshard = reshard.0000000014
482
2017-10-02 15:02:39.932140 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000020
483
2017-10-02 15:02:39.940246 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000021
484
2017-10-02 15:02:39.940626 7fce844ba700 20 proceeding logshard = reshard.0000000015
485
2017-10-02 15:02:39.947949 7fce844ba700  2 object expiration: stop
486
2017-10-02 15:02:39.951058 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000022
487
2017-10-02 15:02:39.959271 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000023
488
2017-10-02 15:02:39.967241 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000024
489
2017-10-02 15:02:39.975455 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000025
490
2017-10-02 15:02:39.984278 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000026
491
2017-10-02 15:02:39.992328 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000027
492
2017-10-02 15:02:39.999937 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000028
493
2017-10-02 15:02:40.034538 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000029
494
2017-10-02 15:02:40.042493 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000030
495
2017-10-02 15:02:40.050255 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000031
496
2017-10-02 15:02:40.059005 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000032
497
2017-10-02 15:02:40.065970 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000033
498
2017-10-02 15:02:40.073446 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000034
499
2017-10-02 15:02:40.082248 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000035
500
2017-10-02 15:02:40.090893 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000036
501
2017-10-02 15:02:40.100179 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000037
502
2017-10-02 15:02:40.108531 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000038
503
2017-10-02 15:02:40.117246 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000039
504
2017-10-02 15:02:40.123850 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000040
505
2017-10-02 15:02:40.131482 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000041
506
2017-10-02 15:02:40.139186 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000042
507
2017-10-02 15:02:40.147111 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000043
508
2017-10-02 15:02:40.155559 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000044
509
2017-10-02 15:02:40.164080 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000045
510
2017-10-02 15:02:40.171653 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000046
511
2017-10-02 15:02:40.183795 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000047
512
2017-10-02 15:02:40.193556 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000048
513
2017-10-02 15:02:40.201827 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000049
514
2017-10-02 15:02:40.209842 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000050
515
2017-10-02 15:02:40.221202 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000051
516
2017-10-02 15:02:40.229971 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000052
517
2017-10-02 15:02:40.237389 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000053
518
2017-10-02 15:02:40.244693 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000054
519
2017-10-02 15:02:40.252030 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000055
520
2017-10-02 15:02:40.259541 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000056
521
2017-10-02 15:02:40.266482 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000057
522
2017-10-02 15:02:40.272126 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000058
523
2017-10-02 15:02:40.278819 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000059
524
2017-10-02 15:02:40.286233 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000060
525
2017-10-02 15:02:40.293359 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000061
526
2017-10-02 15:02:40.300999 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000062
527
2017-10-02 15:02:40.313924 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000063
528
2017-10-02 15:02:40.321276 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000064
529
2017-10-02 15:02:40.329002 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000065
530
2017-10-02 15:02:40.336957 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000066
531
2017-10-02 15:02:40.347802 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000067
532
2017-10-02 15:02:40.356861 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000068
533
2017-10-02 15:02:40.364545 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000069
534
2017-10-02 15:02:40.373646 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000070
535
2017-10-02 15:02:40.380698 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000071
536
2017-10-02 15:02:40.389566 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000072
537
2017-10-02 15:02:40.396041 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000073
538
2017-10-02 15:02:40.403976 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000074
539
2017-10-02 15:02:40.410599 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000075
540
2017-10-02 15:02:40.417743 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000076
541
2017-10-02 15:02:40.424978 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000077
542
2017-10-02 15:02:40.432067 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000078
543
2017-10-02 15:02:40.440238 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000079
544
2017-10-02 15:02:40.447890 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000080
545
2017-10-02 15:02:40.454985 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000081
546
2017-10-02 15:02:40.462615 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000082
547
2017-10-02 15:02:40.470613 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000083
548
2017-10-02 15:02:40.477986 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000084
549
2017-10-02 15:02:40.485846 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000085
550
2017-10-02 15:02:40.494925 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000086
551
2017-10-02 15:02:40.501988 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000087
552
2017-10-02 15:02:40.509930 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000088
553
2017-10-02 15:02:40.517013 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000089
554
2017-10-02 15:02:40.526818 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000090
555
2017-10-02 15:02:40.534622 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000091
556
2017-10-02 15:02:40.543256 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000092
557
2017-10-02 15:02:40.549971 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000093
558
2017-10-02 15:02:40.556459 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000094
559
2017-10-02 15:02:40.565297 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000095
560
2017-10-02 15:02:40.573042 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000096
561
2017-10-02 15:02:40.579414 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000097
562
2017-10-02 15:02:40.586876 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000098
563
2017-10-02 15:02:40.593970 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000099
564
2017-10-02 15:02:40.601411 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000100
565
2017-10-02 15:02:40.609212 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000101
566
2017-10-02 15:02:40.618582 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000102
567
2017-10-02 15:02:40.628529 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000103
568
2017-10-02 15:02:40.636731 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000104
569
2017-10-02 15:02:40.643559 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000105
570
2017-10-02 15:02:40.650913 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000106
571
2017-10-02 15:02:40.659664 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000107
572
2017-10-02 15:02:40.666283 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000108
573
2017-10-02 15:02:40.676757 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000109
574
2017-10-02 15:02:40.685324 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000110
575
2017-10-02 15:02:40.692618 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000111
576
2017-10-02 15:02:40.701160 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000112
577
2017-10-02 15:02:40.707812 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000113
578
2017-10-02 15:02:40.714679 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000114
579
2017-10-02 15:02:40.722360 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000115
580
2017-10-02 15:02:40.729379 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000116
581
2017-10-02 15:02:40.736834 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000117
582
2017-10-02 15:02:40.745126 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000118
583
2017-10-02 15:02:40.753123 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000119
584
2017-10-02 15:02:40.764192 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000120
585
2017-10-02 15:02:40.771209 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000121
586
2017-10-02 15:02:40.778924 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000122
587
2017-10-02 15:02:40.785463 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000123
588
2017-10-02 15:02:40.792048 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000124
589
2017-10-02 15:02:40.800273 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000125
590
2017-10-02 15:02:40.808506 7fce9bce9700 20 proceeding shard = obj_delete_at_hint.0000000126
591
2017-10-02 15:02:40.814620 7fce9bce9700  2 object expiration: stop
592
2017-10-02 15:02:55.295285 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
593
2017-10-02 15:03:17.295405 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
594
2017-10-02 15:03:39.295544 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
595
2017-10-02 15:04:01.295659 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
596
2017-10-02 15:04:23.295761 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
597
2017-10-02 15:04:45.295884 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
598
2017-10-02 15:05:07.296004 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
599
2017-10-02 15:05:29.296102 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
600
2017-10-02 15:05:51.296221 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
601
2017-10-02 15:06:13.296359 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
602
2017-10-02 15:06:35.296485 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
603
2017-10-02 15:06:57.296605 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
604
2017-10-02 15:07:19.296730 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
605
2017-10-02 15:07:24.375473 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246410 cookie 217660536320 notifier 266445 bl.length()=695
606
2017-10-02 15:07:24.375522 7fce9dced700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 info.flags=0x17
607
2017-10-02 15:07:24.375529 7fce9dced700 10 moving default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 to cache LRU end
608
2017-10-02 15:07:24.375539 7fce9dced700 10 updating xattr: name=user.rgw.acl bl.length()=163
609
2017-10-02 15:07:24.375542 7fce9dced700 10 updating xattr: name=user.rgw.idtag bl.length()=0
610
2017-10-02 15:07:24.381507 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246408 cookie 217660532864 notifier 266445 bl.length()=419
611
2017-10-02 15:07:24.381548 7fce9dced700 10 cache put: name=default.rgw.meta+root+image-net info.flags=0x18
612
2017-10-02 15:07:24.381559 7fce9dced700 10 adding default.rgw.meta+root+image-net to cache LRU end
613
2017-10-02 15:07:24.381561 7fce9dced700 10 appending xattr: name=user.rgw.acl bl.length()=246
614
2017-10-02 15:07:24.388891 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246411 cookie 217660536320 notifier 266445 bl.length()=395
615
2017-10-02 15:07:24.388920 7fce9dced700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 info.flags=0x18
616
2017-10-02 15:07:24.388927 7fce9dced700 10 moving default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266445.1 to cache LRU end
617
2017-10-02 15:07:24.388930 7fce9dced700 10 appending xattr: name=user.rgw.acl bl.length()=163
618
2017-10-02 15:07:24.402742 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246409 cookie 217660532864 notifier 266445 bl.length()=619
619
2017-10-02 15:07:24.402779 7fce9dced700 10 cache put: name=default.rgw.meta+root+image-net info.flags=0x17
620
2017-10-02 15:07:24.402784 7fce9dced700 10 moving default.rgw.meta+root+image-net to cache LRU end
621
2017-10-02 15:07:24.402791 7fce9dced700 10 updating xattr: name=user.rgw.acl bl.length()=246
622
2017-10-02 15:07:24.402795 7fce9dced700 10 updating xattr: name=user.rgw.idtag bl.length()=0
623
2017-10-02 15:07:24.409222 7fce9dced700 10 RGWWatcher::handle_notify()  notify_id 11918534246412 cookie 217660536320 notifier 266445 bl.length()=764
624
2017-10-02 15:07:24.409251 7fce9dced700 10 cache put: name=default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266391.1 info.flags=0x17
625
2017-10-02 15:07:24.409256 7fce9dced700 10 moving default.rgw.meta+root+.bucket.meta.image-net:69d2fd65-fcf9-461b-865f-3dbb053803c4.266391.1 to cache LRU end
626
2017-10-02 15:07:24.409261 7fce9dced700 10 updating xattr: name=user.rgw.acl bl.length()=163
627
2017-10-02 15:07:24.409263 7fce9dced700 10 updating xattr: name=user.rgw.idtag bl.length()=0
628
2017-10-02 15:07:39.656088 7fce834b8700  2 keystone revoke thread: start
629
2017-10-02 15:07:39.656115 7fce834b8700  0 ERROR: keystone revocation processing returned error r=-22
630
2017-10-02 15:07:41.296844 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
631
2017-10-02 15:08:03.296970 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
632
2017-10-02 15:08:25.297059 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
633
2017-10-02 15:08:47.297193 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
634
2017-10-02 15:09:09.297330 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
635
2017-10-02 15:09:31.297451 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
636
2017-10-02 15:09:53.297579 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
637
2017-10-02 15:10:15.297693 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
638
2017-10-02 15:10:37.297812 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start
639
2017-10-02 15:10:59.297944 7fce9d4ec700  2 RGWDataChangesLog::ChangesRenewThread: start