Rgw sync agent architecture¶

RGW Data sync
Current scheme:

1. full sync (per shard)
   1. list all buckets
   2. for each bucket in current shard
      1. read bucket marker
      2. sync each object
         1. if failed, add to list to retry later (put in replica log later)
      3. when done with bucket instance, update replica log on destination zone
         1. bucket name
         2. bucket marker (from start of sync)
         3. list of objects to retry
   3. when done with all buckets in a shard,
      1. update replica log on destination zone with
         1. shard number
         2. list of bucket instances to retry
2. incremental sync
   1. look up data log shard in replica log to get buckets that need retrying, and current marker
   2. read data log shard to get changed buckets, starting at marker from replica log
   3. for each bucket
      1. look up objects that need retrying in the replica log
      2. read the bucket instance log for the bucket starting at marker from replica log
      3. sync object using the same method as full sync
      4. if syncing an object fails, add it to a list to retry
      5. update bucket instance replica log with last marker read and list of objects to retry
   4. once data log shard is done, update replica log for that shard with
      1. new marker
      2. list of bucket instances to retry

Suggested scheme:

1. sync initialialization
   1. for each shard:
      1. read marker and buckets to retry for shard’s replica log
         1. if no marker, check shard_id->buckets index
            1. if that doesn’t exist yet, list all buckets and create shard_id -> buckets index
      2. for each bucket incurrent shard
         1. if bucket doesn’t have entry in replica log:
            1. add to replica log, mark for full sync
         2. if bucket exists in replica log, go to (4)(b)
2. incremental sync
   1. look up data log shard in replica log to get buckets that need retrying, and current marker
   2. check type of sync for any buckets that need to be retried, and generate the list of objects based on that
      1. full -> list all objects in bucket
      2. incremental -> read data log as usual
   3. for each bucket
      1. list objects in bucket that need to be synced based on sync type, from full list, or data log
      2. read the bucket instance log for the bucket starting at marker from replica log
      3. sync object using the same method as full sync
      4. if syncing an object fails, add it to a list to retry
      5. update bucket instance replica log with last marker read and list of objects to retry
   4. once data log shard is done, update replica log for that shard with
      1. new marker
      2. list of bucket instances to retry

May need to be careful about updating with an empty marker, if e.g. lots of objects were uploaded before data logging was enabled. Perhaps use ‘ ‘ as the marker in that case, since it’s before all markers the gateway will generate.