Actions
Bug #42429
openLC process falls into an infinite loop during processing transition
% Done:
0%
Source:
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Description
If two LC processes tries to transition the same object by chance, one of them falls into an infinite loop.
The following logs shows what happened when two lc processes conflict at the myobjects_1815916180 object.
2019-10-19 17:28:29.727 7ff8f6f74700 20 lifecycle: bucket_lc_process(): key=lifecycle/myobjects_1815916180 2019-10-19 17:28:29.727 7ff8f6f74700 20 check(): key=lifecycle/myobjects_1815916180: current version, skipping 2019-10-19 17:28:29.727 7ff8f6f74700 20 obj_has_expired(): mtime=2019-10-19 16:10:21.0.480875s days=1 base_time=2019-10-19 17:28:29.728551 timediff=4688.73 cmp=600 2019-10-19 17:28:29.727 7ff8f6f74700 20 check(): key=lifecycle/myobjects_1815916180: is_expired=1 2019-10-19 17:28:29.727 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.942 7ff8f6f74700 10 manifest: total_size = 939000 2019-10-19 17:28:29.942 7ff8f6f74700 20 get_obj_state: setting s->obj_tag to 11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3381854.29704537 2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.acl 2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.content_type 2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.etag 2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.idtag 2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.manifest 2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.pg_ver 2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.source_zone 2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.tail_tag 2019-10-19 17:28:29.942 7ff8f6f74700 20 max_chunk_size=4194304 2019-10-19 17:28:29.942 7ff8f6f74700 20 max_chunk_size=4194304 2019-10-19 17:28:29.942 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.942 7ff8f6f74700 20 max_chunk_size=4194304 2019-10-19 17:28:29.942 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.942 7ff8f6f74700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test 2019-10-19 17:28:29.942 7ff8f6f74700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000 2019-10-19 17:28:29.943 7ff8f6f74700 20 rados->read r=0 bl.length=0 2019-10-19 17:28:29.943 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.943 7ff8f6f74700 20 max_chunk_size=4194304 2019-10-19 17:28:29.943 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.943 7ff8f6f74700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test 2019-10-19 17:28:29.943 7ff8f6f74700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000 2019-10-19 17:28:29.944 7ff8f6f74700 20 rados->read r=0 bl.length=0 2019-10-19 17:28:29.944 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.944 7ff8f6f74700 20 max_chunk_size=4194304 2019-10-19 17:28:29.944 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.944 7ff8f6f74700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test 2019-10-19 17:28:29.944 7ff8f6f74700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000 2019-10-19 17:28:29.944 7ff8f6f74700 20 rados->read r=0 bl.length=0 2019-10-19 17:28:29.944 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.944 7ff8f6f74700 20 max_chunk_size=4194304 2019-10-19 17:28:29.944 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.944 7ff8f6f74700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test 2019-10-19 17:28:29.944 7ff8f6f74700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000 2019-10-19 17:28:29.945 7ff8f6f74700 20 rados->read r=0 bl.length=0 2019-10-19 17:28:29.945 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.945 7ff8f6f74700 20 max_chunk_size=4194304 2019-10-19 17:28:29.945 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.945 7ff8f6f74700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test 2019-10-19 17:28:29.945 7ff8f6f74700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000 2019-10-19 17:28:29.946 7ff8f6f74700 20 rados->read r=0 bl.length=0 2019-10-19 17:28:29.946 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.946 7ff8f6f74700 20 max_chunk_size=4194304 2019-10-19 17:28:29.946 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0 2019-10-19 17:28:29.946 7ff8f6f74700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test 2019-10-19 17:28:29.946 7ff8f6f74700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000 2019-10-19 17:28:29.947 7ff8f6f74700 20 rados->read r=0 bl.length=0 …(infinite loop)
2019-10-19 17:28:29.719 7f17c4f4c700 20 lifecycle: bucket_lc_process(): key=lifecycle/myobjects_1815916180 2019-10-19 17:28:29.719 7f17c4f4c700 20 check(): key=lifecycle/myobjects_1815916180: current version, skipping 2019-10-19 17:28:29.719 7f17c4f4c700 20 obj_has_expired(): mtime=2019-10-19 16:10:21.0.480875s days=1 base_time=2019-10-19 17:28:29.720686 timediff=4688.72 cmp=600 2019-10-19 17:28:29.719 7f17c4f4c700 20 check(): key=lifecycle/myobjects_1815916180: is_expired=1 2019-10-19 17:28:29.719 7f17c4f4c700 20 get_obj_state: rctx=0x7f17c4f48f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x555665f1b1a0 s->prefetch_data=0 2019-10-19 17:28:29.720 7f17c4f4c700 10 manifest: total_size = 939000 2019-10-19 17:28:29.720 7f17c4f4c700 20 get_obj_state: setting s->obj_tag to 11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3381854.29704537 2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.acl 2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.content_type 2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.etag 2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.idtag 2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.manifest 2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.pg_ver 2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.source_zone 2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.tail_tag 2019-10-19 17:28:29.720 7f17c4f4c700 20 max_chunk_size=4194304 2019-10-19 17:28:29.720 7f17c4f4c700 20 max_chunk_size=4194304 2019-10-19 17:28:29.720 7f17c4f4c700 20 get_obj_state: rctx=0x7f17c4f48f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x555665f1b1a0 s->prefetch_data=0 2019-10-19 17:28:29.720 7f17c4f4c700 20 max_chunk_size=4194304 2019-10-19 17:28:29.720 7f17c4f4c700 20 get_obj_state: rctx=0x7f17c4f48f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x555665f1b1a0 s->prefetch_data=0 2019-10-19 17:28:29.720 7f17c4f4c700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test 2019-10-19 17:28:29.720 7f17c4f4c700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000 2019-10-19 17:28:29.726 7f17c4f4c700 20 rados->read r=0 bl.length=939000 2019-10-19 17:28:29.727 7f17c4f4c700 20 max_chunk_size=4194304 2019-10-19 17:28:29.753 7f17c4f4c700 20 get_obj_state: rctx=0x7f17c4f48f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x555665f1b1a0 s->prefetch_data=0 2019-10-19 17:28:29.753 7f17c4f4c700 20 get_obj_state: rctx=0x7f17c4f48f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x555665f1b1a0 s->prefetch_data=0 2019-10-19 17:28:29.753 7f17c4f4c700 10 setting object write_tag=_FRx-D5MQFpyGyOUGKVzI_enI8-TBAk9 2019-10-19 17:28:29.753 7f17c4f4c700 20 reading from default.rgw.meta:root:.bucket.meta.bucket-lc939:11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791 2019-10-19 17:28:29.753 7f17c4f4c700 20 get_system_obj_state: rctx=0x7f17c4f47100 obj=default.rgw.meta:root:.bucket.meta.bucket-lc939:11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791 state=0x5556666b76c0 s->prefetch_data=0 2019-10-19 17:28:29.753 7f17c4f4c700 10 cache get: name=default.rgw.meta+root+.bucket.meta.bucket-lc939:11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791 : hit (requested=0x16, cached=0x17) 2019-10-19 17:28:29.753 7f17c4f4c700 20 get_system_obj_state: s->obj_tag was set empty 2019-10-19 17:28:29.753 7f17c4f4c700 10 cache get: name=default.rgw.meta+root+.bucket.meta.bucket-lc939:11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791 : hit (requested=0x11, cached=0x17) 2019-10-19 17:28:29.753 7f17c4f4c700 20 bucket index object: .dir.11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791.0 2019-10-19 17:28:29.782 7f17c4f4c700 2 TRANSITIONED:bucket-lc939[11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791]:lifecycle/myobjects_1815916180 -> STANDARD_IA 2019-10-19 17:28:29.782 7f17c4f4c700 20 processed:bucket-lc939[11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791]:lifecycle/myobjects_1815916180
Actions