Bug #15480
closedMultisite sync stopped/hung after uploading a multipart object
0%
Description
Configured active-active multisite with two zones. Verified that the buckets and objects were syncing. Create a multipart object on the master zone. But the object did not get synced to the secondary zone. Any subsequent object creations are being queued and the sync status shows as "syncing"
Seen on ceph-radosgw-10.1.1-1.el7cp.x86_64
Steps to Reproduce:
1. Configure active active multisite clusters with two zones in it.
2. Create objects/buckets in each zone and verify if they are syncing
3. Try a multipart upload on one of the zones. I tried a 1.5 GB file upload on master zone and then another multipart file from secondary zone.
4. The sync status is hung for almost an hour.
- radosgw-admin sync status
realm 4e00a610-36e9-43d0-803e-4001442b8232 (earth)
zonegroup e66e1293-e63b-4afe-9dad-3397647dfb03 (us)
zone acadcc66-10b9-4829-b8e2-306c0048bff5 (us-1)
metadata sync no sync (zone is master)
data sync source: 001da65b-c3a8-42e2-a1ce-79cacefbace2 (us-2)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 2 shards
- radosgw-admin sync status --rgw-zone=us-2
realm 4e00a610-36e9-43d0-803e-4001442b8232 (earth)
zonegroup e66e1293-e63b-4afe-9dad-3397647dfb03 (us)
zone 001da65b-c3a8-42e2-a1ce-79cacefbace2 (us-2)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is behind on 5 shards
oldest incremental change not applied: 2016-04-12 09:28:08.0.696735s
data sync source: acadcc66-10b9-4829-b8e2-306c0048bff5 (us-1)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 2 shards
oldest incremental change not applied: 2016-04-12 08:39:55.0.614701s
Any subsequent creation of buckets or objects are not syncing.
On master:
]# radosgw-admin bucket list
[
"container3",
"container5",
"new-bucket",
"container2",
"bigbucket",
"container4",
"my-new-bucket",
"container"
]
On the peer zone:
- radosgw-admin bucket list
[
"container3",
"bigbucket",
"container4",
"my-new-bucket",
"container"
]
- swift -A http://rgw1:8080/auth/1.0 -U test-user:swift -K 'kzmbCQgR3L5CqjQmvjatXLjeZi1Ss8RFlWLGu1Vj' list bigbucket
big.txt
- swift -A http://rgw2:8080/auth/1.0 -U test-user:swift -K 'kzmbCQgR3L5CqjQmvjatXLjeZi1Ss8RFlWLGu1Vj' list bigbucket
f22.iso
I don't find any sync errors in the rgw logs.