Project

General

Profile

Actions
Copy link

Bug #42429

open

LC process falls into an infinite loop during processing transition

Added by Ilsoo Byun over 4 years ago. Updated over 4 years ago.

Status:
Fix Under Review
Priority:
Normal
Assignee:
Matt Benjamin
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
31214
Crash signature (v1):
Crash signature (v2):

Description

If two LC processes tries to transition the same object by chance, one of them falls into an infinite loop.
The following logs shows what happened when two lc processes conflict at the myobjects_1815916180 object.

2019-10-19 17:28:29.727 7ff8f6f74700 20 lifecycle: bucket_lc_process(): key=lifecycle/myobjects_1815916180
2019-10-19 17:28:29.727 7ff8f6f74700 20 check(): key=lifecycle/myobjects_1815916180: current version, skipping
2019-10-19 17:28:29.727 7ff8f6f74700 20 obj_has_expired(): mtime=2019-10-19 16:10:21.0.480875s days=1 base_time=2019-10-19 17:28:29.728551 timediff=4688.73 cmp=600
2019-10-19 17:28:29.727 7ff8f6f74700 20 check(): key=lifecycle/myobjects_1815916180: is_expired=1
2019-10-19 17:28:29.727 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.942 7ff8f6f74700 10 manifest: total_size = 939000
2019-10-19 17:28:29.942 7ff8f6f74700 20 get_obj_state: setting s->obj_tag to 11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3381854.29704537
2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.acl
2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.content_type
2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.etag
2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.idtag
2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.manifest
2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.pg_ver
2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.source_zone
2019-10-19 17:28:29.942 7ff8f6f74700 20 Read xattr: user.rgw.tail_tag
2019-10-19 17:28:29.942 7ff8f6f74700 20 max_chunk_size=4194304
2019-10-19 17:28:29.942 7ff8f6f74700 20 max_chunk_size=4194304
2019-10-19 17:28:29.942 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.942 7ff8f6f74700 20 max_chunk_size=4194304
2019-10-19 17:28:29.942 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.942 7ff8f6f74700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test
2019-10-19 17:28:29.942 7ff8f6f74700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000
2019-10-19 17:28:29.943 7ff8f6f74700 20 rados->read r=0 bl.length=0
2019-10-19 17:28:29.943 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.943 7ff8f6f74700 20 max_chunk_size=4194304
2019-10-19 17:28:29.943 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.943 7ff8f6f74700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test
2019-10-19 17:28:29.943 7ff8f6f74700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000
2019-10-19 17:28:29.944 7ff8f6f74700 20 rados->read r=0 bl.length=0
2019-10-19 17:28:29.944 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.944 7ff8f6f74700 20 max_chunk_size=4194304
2019-10-19 17:28:29.944 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.944 7ff8f6f74700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test
2019-10-19 17:28:29.944 7ff8f6f74700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000
2019-10-19 17:28:29.944 7ff8f6f74700 20 rados->read r=0 bl.length=0
2019-10-19 17:28:29.944 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.944 7ff8f6f74700 20 max_chunk_size=4194304
2019-10-19 17:28:29.944 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.944 7ff8f6f74700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test
2019-10-19 17:28:29.944 7ff8f6f74700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000
2019-10-19 17:28:29.945 7ff8f6f74700 20 rados->read r=0 bl.length=0
2019-10-19 17:28:29.945 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.945 7ff8f6f74700 20 max_chunk_size=4194304
2019-10-19 17:28:29.945 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.945 7ff8f6f74700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test
2019-10-19 17:28:29.945 7ff8f6f74700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000
2019-10-19 17:28:29.946 7ff8f6f74700 20 rados->read r=0 bl.length=0
2019-10-19 17:28:29.946 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.946 7ff8f6f74700 20 max_chunk_size=4194304
2019-10-19 17:28:29.946 7ff8f6f74700 20 get_obj_state: rctx=0x7ff8f6f70f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x559ee08e8ea0 s->prefetch_data=0
2019-10-19 17:28:29.946 7ff8f6f74700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test
2019-10-19 17:28:29.946 7ff8f6f74700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000
2019-10-19 17:28:29.947 7ff8f6f74700 20 rados->read r=0 bl.length=0
…(infinite loop)


2019-10-19 17:28:29.719 7f17c4f4c700 20 lifecycle: bucket_lc_process(): key=lifecycle/myobjects_1815916180
2019-10-19 17:28:29.719 7f17c4f4c700 20 check(): key=lifecycle/myobjects_1815916180: current version, skipping
2019-10-19 17:28:29.719 7f17c4f4c700 20 obj_has_expired(): mtime=2019-10-19 16:10:21.0.480875s days=1 base_time=2019-10-19 17:28:29.720686 timediff=4688.72 cmp=600
2019-10-19 17:28:29.719 7f17c4f4c700 20 check(): key=lifecycle/myobjects_1815916180: is_expired=1
2019-10-19 17:28:29.719 7f17c4f4c700 20 get_obj_state: rctx=0x7f17c4f48f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x555665f1b1a0 s->prefetch_data=0
2019-10-19 17:28:29.720 7f17c4f4c700 10 manifest: total_size = 939000
2019-10-19 17:28:29.720 7f17c4f4c700 20 get_obj_state: setting s->obj_tag to 11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3381854.29704537
2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.acl
2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.content_type
2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.etag
2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.idtag
2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.manifest
2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.pg_ver
2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.source_zone
2019-10-19 17:28:29.720 7f17c4f4c700 20 Read xattr: user.rgw.tail_tag
2019-10-19 17:28:29.720 7f17c4f4c700 20 max_chunk_size=4194304
2019-10-19 17:28:29.720 7f17c4f4c700 20 max_chunk_size=4194304
2019-10-19 17:28:29.720 7f17c4f4c700 20 get_obj_state: rctx=0x7f17c4f48f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x555665f1b1a0 s->prefetch_data=0
2019-10-19 17:28:29.720 7f17c4f4c700 20 max_chunk_size=4194304
2019-10-19 17:28:29.720 7f17c4f4c700 20 get_obj_state: rctx=0x7f17c4f48f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x555665f1b1a0 s->prefetch_data=0
2019-10-19 17:28:29.720 7f17c4f4c700 20 state for obj=bucket-lc939:lifecycle/myobjects_1815916180 is not atomic, not appending atomic test
2019-10-19 17:28:29.720 7f17c4f4c700 20 rados->read obj-ofs=0 read_ofs=0 read_len=939000
2019-10-19 17:28:29.726 7f17c4f4c700 20 rados->read r=0 bl.length=939000
2019-10-19 17:28:29.727 7f17c4f4c700 20 max_chunk_size=4194304
2019-10-19 17:28:29.753 7f17c4f4c700 20 get_obj_state: rctx=0x7f17c4f48f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x555665f1b1a0 s->prefetch_data=0
2019-10-19 17:28:29.753 7f17c4f4c700 20 get_obj_state: rctx=0x7f17c4f48f98 obj=bucket-lc939:lifecycle/myobjects_1815916180 state=0x555665f1b1a0 s->prefetch_data=0
2019-10-19 17:28:29.753 7f17c4f4c700 10 setting object write_tag=_FRx-D5MQFpyGyOUGKVzI_enI8-TBAk9
2019-10-19 17:28:29.753 7f17c4f4c700 20 reading from default.rgw.meta:root:.bucket.meta.bucket-lc939:11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791
2019-10-19 17:28:29.753 7f17c4f4c700 20 get_system_obj_state: rctx=0x7f17c4f47100 obj=default.rgw.meta:root:.bucket.meta.bucket-lc939:11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791 state=0x5556666b76c0 s->prefetch_data=0
2019-10-19 17:28:29.753 7f17c4f4c700 10 cache get: name=default.rgw.meta+root+.bucket.meta.bucket-lc939:11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791 : hit (requested=0x16, cached=0x17)
2019-10-19 17:28:29.753 7f17c4f4c700 20 get_system_obj_state: s->obj_tag was set empty
2019-10-19 17:28:29.753 7f17c4f4c700 10 cache get: name=default.rgw.meta+root+.bucket.meta.bucket-lc939:11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791 : hit (requested=0x11, cached=0x17)
2019-10-19 17:28:29.753 7f17c4f4c700 20  bucket index object: .dir.11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791.0
2019-10-19 17:28:29.782 7f17c4f4c700  2 TRANSITIONED:bucket-lc939[11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791]:lifecycle/myobjects_1815916180 -> STANDARD_IA
2019-10-19 17:28:29.782 7f17c4f4c700 20 processed:bucket-lc939[11fd7df3-e916-4ac4-af14-7dca2dd1ee0d.3402367.3791]:lifecycle/myobjects_1815916180
Actions
Copy link

Also available in: Atom PDF