Project

General

Profile

Bug #21990

multisite: race between sync of bucket and bucket instance metadata

Added by Casey Bodley about 2 months ago. Updated about 2 months ago.

Status:
Pending Backport
Priority:
High
Assignee:
Target version:
-
Start date:
10/31/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

As part of https://github.com/ceph/ceph/pull/12704 for http://tracker.ceph.com/issues/17932, a new 'placement_rule' argument was added to rgw_link_bucket(). rgw_link_bucket() is called by RGWBucketMetadataHandler::put() to make the bucket entrypoint visible to its owner. But the placement_rule comes from the bucket instance rather than the bucket entrypoint, so RGWBucketMetadataHandler::put() tries to read it from the bucket instance.

This adds a dependency between the bucket entrypoint and the bucket instance that multisite's metadata sync does not account for. If the bucket entrypoint for a new bucket finishes syncing before its bucket instance, RGWBucketMetadataHandler::put() fails with -ENOENT because the bucket instance doesn't exist yet:

2017-10-31 17:05:04.667230 7f8d7cc41700 20 rgw_get_bucket_info: bucket instance: new_buc_0[bf0750fa-3531-4110-a224-2df15c2c445f.174148.1])
2017-10-31 17:05:04.667234 7f8d7cc41700 20 reading from us-west.rgw.meta:root:.bucket.meta.new_buc_0:bf0750fa-3531-4110-a224-2df15c2c445f.174148.1
2017-10-31 17:05:04.667237 7f8d7cc41700 20 get_system_obj_state: rctx=0x7f8d7cc3eaa0 obj=us-west.rgw.meta:root:.bucket.meta.new_buc_0:bf0750fa-3531-4110-a224-2df15c2c445f.174148.1 state=0x564b7b8e32a0 s->prefetch_data=0
2017-10-31 17:05:04.667243 7f8d7cc41700 10 cache get: name=us-west.rgw.meta+root+.bucket.meta.new_buc_0:bf0750fa-3531-4110-a224-2df15c2c445f.174148.1 : miss
2017-10-31 17:05:04.667845 7f8d7cc41700 10 cache put: name=us-west.rgw.meta+root+.bucket.meta.new_buc_0:bf0750fa-3531-4110-a224-2df15c2c445f.174148.1 info.flags=0x0
2017-10-31 17:05:04.667856 7f8d7cc41700 10 adding us-west.rgw.meta+root+.bucket.meta.new_buc_0:bf0750fa-3531-4110-a224-2df15c2c445f.174148.1 to cache LRU end
2017-10-31 17:05:04.667897 7f8d7cc41700  0 meta sync: ERROR: can't store key: bucket:new_buc_0 ret=-2
2017-10-31 17:05:04.667929 7f8d79c3b700 20 cr:s=0x564b7d2b3260:op=0x564b79f33800:19RGWMetaStoreEntryCR: operate()
2017-10-31 17:05:04.667936 7f8d79c3b700 20 cr:s=0x564b7d2b3260:op=0x564b79f33800:19RGWMetaStoreEntryCR: operate() returned r=-2
2017-10-31 17:05:04.667940 7f8d79c3b700 20 cr:s=0x564b7d2b3260:op=0x564b7b67ac00:24RGWMetaSyncSingleEntryCR: operate()
2017-10-31 17:05:04.667941 7f8d79c3b700 20 cr:s=0x564b7d2b3260:op=0x564b7b67ac00:24RGWMetaSyncSingleEntryCR: operate() returned r=-2

Related issues

Copied to rgw - Backport #22020: luminous: multisite: race between sync of bucket and bucket instance metadata In Progress

History

#1 Updated by Casey Bodley about 2 months ago

  • Status changed from Verified to Need Review

#2 Updated by Ken Dreyer about 2 months ago

  • Status changed from Need Review to Pending Backport

#3 Updated by Nathan Cutler about 1 month ago

  • Copied to Backport #22020: luminous: multisite: race between sync of bucket and bucket instance metadata added

Also available in: Atom PDF