Ceph » rgw

Configured active-active multisite with two zones. Verified that the buckets and objects were syncing. Create a multipart object on the master zone. But the object did not get synced to the secondary zone. Any subsequent object creations are being queued and the sync status shows as "syncing"

Seen on ceph-radosgw-10.1.1-1.el7cp.x86_64

Steps to Reproduce:
1. Configure active active multisite clusters with two zones in it.
2. Create objects/buckets in each zone and verify if they are syncing
3. Try a multipart upload on one of the zones. I tried a 1.5 GB file upload on master zone and then another multipart file from secondary zone.
4. The sync status is hung for almost an hour.

1. radosgw-admin sync status
   realm 4e00a610-36e9-43d0-803e-4001442b8232 (earth)
   zonegroup e66e1293-e63b-4afe-9dad-3397647dfb03 (us)
   zone acadcc66-10b9-4829-b8e2-306c0048bff5 (us-1)
   metadata sync no sync (zone is master)
   data sync source: 001da65b-c3a8-42e2-a1ce-79cacefbace2 (us-2)
   syncing
   full sync: 0/128 shards
   incremental sync: 128/128 shards
   data is behind on 2 shards

1. radosgw-admin sync status --rgw-zone=us-2
   realm 4e00a610-36e9-43d0-803e-4001442b8232 (earth)
   zonegroup e66e1293-e63b-4afe-9dad-3397647dfb03 (us)
   zone 001da65b-c3a8-42e2-a1ce-79cacefbace2 (us-2)
   metadata sync syncing
   full sync: 0/64 shards
   incremental sync: 64/64 shards
   metadata is behind on 5 shards
   oldest incremental change not applied: 2016-04-12 09:28:08.0.696735s
   data sync source: acadcc66-10b9-4829-b8e2-306c0048bff5 (us-1)
   syncing
   full sync: 0/128 shards
   incremental sync: 128/128 shards
   data is behind on 2 shards
   oldest incremental change not applied: 2016-04-12 08:39:55.0.614701s

Any subsequent creation of buckets or objects are not syncing.

On master:

]# radosgw-admin bucket list
[
"container3",
"container5",
"new-bucket",
"container2",
"bigbucket",
"container4",
"my-new-bucket",
"container"
]

On the peer zone:

1. radosgw-admin bucket list
   [
   "container3",
   "bigbucket",
   "container4",
   "my-new-bucket",
   "container"
   ]

From master:

1. swift -A http://rgw1:8080/auth/1.0 -U test-user:swift -K 'kzmbCQgR3L5CqjQmvjatXLjeZi1Ss8RFlWLGu1Vj' list bigbucket
   big.txt

From peer:

1. swift -A http://rgw2:8080/auth/1.0 -U test-user:swift -K 'kzmbCQgR3L5CqjQmvjatXLjeZi1Ss8RFlWLGu1Vj' list bigbucket
   f22.iso

I don't find any sync errors in the rgw logs.