Ceph » rgw

Based on https://docs.ceph.com/docs/master/radosgw/multisite/
"Important You must execute metadata operations, such as user creation,
on a host within the master zone. The master zone and the secondary zone
can receive bucket operations, but the secondary zone redirects bucket
operations to the master zone. If the master zone is down, bucket
operations will fail."
it looks like the bucket create operations are redirected to the master,
but a bucket delete is not, so it takes effect only locally on the secondary.

ceph --version
ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable)

I experimented further today to confirm a data loss scenario.

The buckets I experimented with yesterday were empty.
It is possible to delete a bucket only if it is empty, so in the scenario
where a bucket is deleted on the secondary but remains on the primary,
recreating a bucket with the same name on the secondary cannot lead
immediately to data loss.

However, I tried this:

• create bucket1 through dashboard of primary
• create bucket2 through dashboard of secondary
• delete both through dashboard of secondary; both still visible on primary
  (as previously described - bad, but no data loss yet...)
• create duplicate bucket2 through dashboard of secondary
• use s3cmd to add an object to bucket2 through the secondary
  - the object is copied to the original bucket2 on the primary (good)
• use s3cmd to add an object to bucket1 through the primary
  - bucket1 is not recreated on the secondary (very bad)

We now have a silent failure to replicate bucket2 from the primary to the secondary.
Note that the object put into the bucket in each case was a one-line text file.
After a few minutes I did (on the secondary; the primary said the same):

radosgw-admin sync status
          realm 46f6fa65-3cc1-47c4-8184-f0f8a7441097 (_89175832-1dec-45f8-8544-c0b2bfe8f707)
      zonegroup 89175832-1dec-45f8-8544-c0b2bfe8f707 (_89175832-1dec-45f8-8544-c0b2bfe8f707)
           zone 203de9cc-284a-4102-aa87-f0b04ad66130 (store-mb2)
  metadata sync syncing
                full sync: 0/64 shards
                incremental sync: 64/64 shards
                metadata is caught up with master
      data sync source: 581307c1-6f65-44c7-9a88-b4e364dcad22 (store-mb1)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        1 shards are recovering
                        recovering shards: [0]

A little later, I repeated it and got:

radosgw-admin sync status
          realm 46f6fa65-3cc1-47c4-8184-f0f8a7441097 (_89175832-1dec-45f8-8544-c0b2bfe8f707)
      zonegroup 89175832-1dec-45f8-8544-c0b2bfe8f707 (_89175832-1dec-45f8-8544-c0b2bfe8f707)
           zone 203de9cc-284a-4102-aa87-f0b04ad66130 (store-mb2)
  metadata sync syncing
                full sync: 0/64 shards
                incremental sync: 64/64 shards
                metadata is caught up with master
      data sync source: 581307c1-6f65-44c7-9a88-b4e364dcad22 (store-mb1)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

But there was still no copy of bucket1 on the secondary.

We now have a silent failure to replicate bucket2 from the primary to the secondary.

should of course read "failure to replicate bucket1"