Bug #21990
multisite: race between sync of bucket and bucket instance metadata
0%
Description
As part of https://github.com/ceph/ceph/pull/12704 for http://tracker.ceph.com/issues/17932, a new 'placement_rule' argument was added to rgw_link_bucket(). rgw_link_bucket() is called by RGWBucketMetadataHandler::put() to make the bucket entrypoint visible to its owner. But the placement_rule comes from the bucket instance rather than the bucket entrypoint, so RGWBucketMetadataHandler::put() tries to read it from the bucket instance.
This adds a dependency between the bucket entrypoint and the bucket instance that multisite's metadata sync does not account for. If the bucket entrypoint for a new bucket finishes syncing before its bucket instance, RGWBucketMetadataHandler::put() fails with -ENOENT because the bucket instance doesn't exist yet:
2017-10-31 17:05:04.667230 7f8d7cc41700 20 rgw_get_bucket_info: bucket instance: new_buc_0[bf0750fa-3531-4110-a224-2df15c2c445f.174148.1]) 2017-10-31 17:05:04.667234 7f8d7cc41700 20 reading from us-west.rgw.meta:root:.bucket.meta.new_buc_0:bf0750fa-3531-4110-a224-2df15c2c445f.174148.1 2017-10-31 17:05:04.667237 7f8d7cc41700 20 get_system_obj_state: rctx=0x7f8d7cc3eaa0 obj=us-west.rgw.meta:root:.bucket.meta.new_buc_0:bf0750fa-3531-4110-a224-2df15c2c445f.174148.1 state=0x564b7b8e32a0 s->prefetch_data=0 2017-10-31 17:05:04.667243 7f8d7cc41700 10 cache get: name=us-west.rgw.meta+root+.bucket.meta.new_buc_0:bf0750fa-3531-4110-a224-2df15c2c445f.174148.1 : miss 2017-10-31 17:05:04.667845 7f8d7cc41700 10 cache put: name=us-west.rgw.meta+root+.bucket.meta.new_buc_0:bf0750fa-3531-4110-a224-2df15c2c445f.174148.1 info.flags=0x0 2017-10-31 17:05:04.667856 7f8d7cc41700 10 adding us-west.rgw.meta+root+.bucket.meta.new_buc_0:bf0750fa-3531-4110-a224-2df15c2c445f.174148.1 to cache LRU end 2017-10-31 17:05:04.667897 7f8d7cc41700 0 meta sync: ERROR: can't store key: bucket:new_buc_0 ret=-2 2017-10-31 17:05:04.667929 7f8d79c3b700 20 cr:s=0x564b7d2b3260:op=0x564b79f33800:19RGWMetaStoreEntryCR: operate() 2017-10-31 17:05:04.667936 7f8d79c3b700 20 cr:s=0x564b7d2b3260:op=0x564b79f33800:19RGWMetaStoreEntryCR: operate() returned r=-2 2017-10-31 17:05:04.667940 7f8d79c3b700 20 cr:s=0x564b7d2b3260:op=0x564b7b67ac00:24RGWMetaSyncSingleEntryCR: operate() 2017-10-31 17:05:04.667941 7f8d79c3b700 20 cr:s=0x564b7d2b3260:op=0x564b7b67ac00:24RGWMetaSyncSingleEntryCR: operate() returned r=-2
Related issues
History
#1 Updated by Casey Bodley over 6 years ago
- Status changed from 12 to Fix Under Review
#2 Updated by Ken Dreyer over 6 years ago
- Status changed from Fix Under Review to Pending Backport
#3 Updated by Nathan Cutler over 6 years ago
- Copied to Backport #22020: luminous: multisite: race between sync of bucket and bucket instance metadata added
#4 Updated by Nathan Cutler almost 6 years ago
- Status changed from Pending Backport to Resolved