Feature #59593
closedThe capability of resetting an empty bucket to the clean-slate state in multi-site environment
0%
Description
In a multi-site environment, we can sync objects of the same bucket or even different buckets between zones. This serves the purpose of data backup/recovery well. However, if we just want to migrate the objects of a bucket from one zone to another, the current solution seems to be inadequate.
Right now, we can copy the objects of a bucket from one zone (let's say A) to another zone (let's say B) via bucket-level sync. To achieve the purpose of migration, after disabling sync, we need to remove all objects of the bucket from zone A. However, later if we want to migrate the objects back from zone B to zone A, we will have troubles because zone A has updated its max_markers on each shard to reflect the history of object removals. These markers prevent the objects from zone B from migrating back since their markers are "older". Only those objects that were created later will be sync-ed over.
However, if we can reset these markers or delete them on an empty bucket, and, at the same time, ensure all bilogs have been auto-trimmed, making a full-sync from zone B to zone A will bring back all objects in zone B, including those that have been deleted in zone A earlier, thus start the process of another migration. My own prototype that enables this capability of resetting approves that it is viable. Right now, ceph/rgw doesn't seem to have this capability. Therefore, I created this feature request ticket to seek the possibility of introducing this capability into ceph/rgw.
Updated by Casey Bodley 12 months ago
Yixin Jin wrote:
However, later if we want to migrate the objects back from zone B to zone A, we will have troubles because zone A has updated its max_markers on each shard to reflect the history of object removals. These markers prevent the objects from zone B from migrating back since their markers are "older". Only those objects that were created later will be sync-ed over.
i don't think max_markers comes up here. as far as i know, bucket sync only queries the remote max_markers at the beginning of full sync so it knows where to resume incremental sync after the full sync finishes
the reason that zone A wouldn't resync objects from zone B is that it already processed their associated bilog entries and updated its bucket sync status to reflect that. once zone B trims those bilog entries, the only way to resync the objects would be to restart a bucket full sync on zone A
there are two existing ways to force that with radosgw-admin:
bucket sync init --source-zone B --bucket foobucket sync disable --bucket foofollowed bybucket sync enable
Updated by Casey Bodley 12 months ago
- Project changed from Ceph to rgw
- Tags set to multisite
Updated by Yixin Jin 12 months ago
Casey Bodley wrote:
Yixin Jin wrote:
However, later if we want to migrate the objects back from zone B to zone A, we will have troubles because zone A has updated its max_markers on each shard to reflect the history of object removals. These markers prevent the objects from zone B from migrating back since their markers are "older". Only those objects that were created later will be sync-ed over.
i don't think
max_markerscomes up here. as far as i know, bucket sync only queries the remotemax_markersat the beginning of full sync so it knows where to resume incremental sync after the full sync finishesthe reason that zone A wouldn't resync objects from zone B is that it already processed their associated bilog entries and updated its bucket sync status to reflect that. once zone B trims those bilog entries, the only way to resync the objects would be to restart a bucket full sync on zone A
there are two existing ways to force that with radosgw-admin:
bucket sync init --source-zone B --bucket foobucket sync disable --bucket foofollowed bybucket sync enable
I see. I suspect some of my observation during my experiment may be related the availability of bilogs. For example, if I sync objects from zone A to zone B, disable the sync and then delete objects in zone A to conclude the process of migration, if I don't do bilog autotrim right after object removals and if I re-enable symmetrical sync to start the process of migrating objects back to zone A, I will end up having empty bucket in both zone A and zone B. On the other hand, if I do autotrim, re-enable sync followed by "bucket sync init" and "bucket sync run", objects will be copied back from zone B to zone A.