Actions
Bug #22908
open[Multisite] Synchronization works only one way (zone2->zone1)
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rgw
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I have noticed that synchronization stopped working for some reason (but not fully - let me explain):
Everything was ok till 31.01.2018:
➜ ~ s3cmd -c zone1 ls s3://<bucket name>/2018/01/30/20/ | wc -l
18
➜ ~ s3cmd -c zone2 ls s3://<bucket name>/2018/01/30/20/ | wc -l
18
➜ ~
And after that:
➜ ~ s3cmd -c zone1 ls s3://<bucket name>/2018/01/30/21/ | wc -l
18
➜ ~ s3cmd -c zone2 ls s3://<bucket name>/2018/01/30/21/ | wc -l
12
➜ ~
I have the name of the DC in the filename (from where data come from):
Zone1 - master:
2018-01-30 20:15 1757117 s3://<bucket name>/2018/01/30/21/2356202233201122-52891-v1-zone1
2018-01-30 20:16 1755338 s3://<bucket name>/2018/01/30/21/2356407377147077-51725-v1-zone1
2018-01-30 20:31 1795243 s3://<bucket name>/2018/01/30/21/2357138004184386-52607-v1-zone1
2018-01-30 20:16 1766473 s3://<bucket name>/2018/01/30/21/2357153301329742-52479-v1-zone1
2018-01-30 20:31 1835095 s3://<bucket name>/2018/01/30/21/2357342194418114-53850-v1-zone1
2018-01-30 20:16 1749582 s3://<bucket name>/2018/01/30/21/2357549767263837-52026-v1-zone1
2018-01-30 20:47 1740989 s3://<bucket name>/2018/01/30/21/2358073001616294-51939-v1-zone1
2018-01-30 20:31 1841696 s3://<bucket name>/2018/01/30/21/2358088303417457-54688-v1-zone1
2018-01-30 20:47 1713001 s3://<bucket name>/2018/01/30/21/2358276846382849-50000-v1-zone1
2018-01-30 20:31 1792212 s3://<bucket name>/2018/01/30/21/2358484311300704-52251-v1-zone1
2018-01-30 21:03 1430706 s3://<bucket name>/2018/01/30/21/2359008017818455-42080-v1-zone1
2018-01-30 20:47 1725195 s3://<bucket name>/2018/01/30/21/2359022892851188-50959-v1-zone1
2018-01-30 21:03 1443962 s3://<bucket name>/2018/01/30/21/2359211503351068-41784-v1-zone1
2018-01-30 20:47 1747334 s3://<bucket name>/2018/01/30/21/2359418738089062-52037-v1-zone1
2018-01-30 20:35 2556 s3://<bucket name>/2018/01/30/21/2359498340525216-8-v1-zone2
2018-01-30 21:03 1425118 s3://<bucket name>/2018/01/30/21/2359956779752022-41868-v1-zone1
2018-01-30 21:03 1431091 s3://<bucket name>/2018/01/30/21/2360352785119795-42209-v1-zone1
2018-01-30 21:20 2564 s3://<bucket name>/2018/01/30/21/2362228740122179-3-v1-zone2
Zone2 - secondary
2018-01-30 20:16 1755338 s3://<bucket name>/2018/01/30/21/2356407377147077-51725-v1-zone1
2018-01-30 20:31 1795243 s3://<bucket name>/2018/01/30/21/2357138004184386-52607-v1-zone1
2018-01-30 20:16 1766473 s3://<bucket name>/2018/01/30/21/2357153301329742-52479-v1-zone1
2018-01-30 20:31 1835095 s3://<bucket name>/2018/01/30/21/2357342194418114-53850-v1-zone1
2018-01-30 20:16 1749582 s3://<bucket name>/2018/01/30/21/2357549767263837-52026-v1-zone1
2018-01-30 20:47 1740989 s3://<bucket name>/2018/01/30/21/2358073001616294-51939-v1-zone1
2018-01-30 20:31 1841696 s3://<bucket name>/2018/01/30/21/2358088303417457-54688-v1-zone1
2018-01-30 20:31 1792212 s3://<bucket name>/2018/01/30/21/2358484311300704-52251-v1-zone1
2018-01-30 20:47 1725195 s3://<bucket name>/2018/01/30/21/2359022892851188-50959-v1-zone1
2018-01-30 20:35 2556 s3://<bucket name>/2018/01/30/21/2359498340525216-8-v1-zone2
2018-01-30 21:20 2564 s3://<bucket name>/2018/01/30/21/2362228740122179-3-v1-zone2
So that means that I dont have a few files from the zone1 in the zone2. After that date I am not able to see any files from the zone1 in the zone2:
zone2:
2018-01-30 21:38 2594 s3://<bucket name>/2018/01/30/22/2363278763714103-12-v1-zone2
2018-01-30 22:11 2480 s3://<bucket name>/2018/01/30/22/2365288966899244-3-v1-zone2
zone1:
2018-01-30 21:15 1525212 s3://<bucket name>/2018/01/30/22/2359792201857100-44183-v1-zone1
2018-01-30 21:15 1581953 s3://<bucket name>/2018/01/30/22/2359995487978309-46588-v1-zone1
2018-01-30 21:31 1459499 s3://<bucket name>/2018/01/30/22/2360726266479292-43200-v1-zone1
2018-01-30 21:15 1529054 s3://<bucket name>/2018/01/30/22/2360740758774808-45008-v1-zone1
2018-01-30 21:31 1483060 s3://<bucket name>/2018/01/30/22/2360929541234751-44088-v1-zone1
2018-01-30 21:15 1528468 s3://<bucket name>/2018/01/30/22/2361136711431588-45084-v1-zone1
2018-01-30 21:47 1322918 s3://<bucket name>/2018/01/30/22/2361661248467302-39156-v1-zone1
2018-01-30 21:31 1459381 s3://<bucket name>/2018/01/30/22/2361674440853750-43447-v1-zone1
2018-01-30 21:47 1330364 s3://<bucket name>/2018/01/30/22/2361863632474932-39708-v1-zone1
2018-01-30 21:31 1447952 s3://<bucket name>/2018/01/30/22/2362070303351222-42168-v1-zone1
2018-01-30 22:02 964967 s3://<bucket name>/2018/01/30/22/2362596483938629-29084-v1-zone1
2018-01-30 21:47 1323983 s3://<bucket name>/2018/01/30/22/2362608066604117-38788-v1-zone1
2018-01-30 22:02 1011242 s3://<bucket name>/2018/01/30/22/2362796736684101-31161-v1-zone1
2018-01-30 21:47 1312808 s3://<bucket name>/2018/01/30/22/2363003556814073-38029-v1-zone1
2018-01-30 21:38 2594 s3://<bucket name>/2018/01/30/22/2363278763714103-12-v1-zone2
2018-01-30 22:03 985933 s3://<bucket name>/2018/01/30/22/2363541027813764-30649-v1-zone1
2018-01-30 22:02 1005303 s3://<bucket name>/2018/01/30/22/2363936616223993-30624-v1-zone1
2018-01-30 22:11 2480 s3://<bucket name>/2018/01/30/22/2365288966899244-3-v1-zone2
Status is a little bit wired:
zone2 -> zone1 = OK
zone1 -> zone2 = NOT OK
If we take a look at the synchronization status:
zone1:
realm c6055c2e-5ac0-4638-851f-f1051b61d0c2 (platform)
zonegroup 4134640c-d16b-4166-bbd6-987637da469d (prd)
zone 8adfe5fc-65df-4227-9d85-1d0d1e66ac1f (zone1)
metadata sync no sync (zone is master)
data sync source: 6328c6d7-31a5-4d42-8359-1e28689572da (zone2)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 1 shards
zone2:
realm c6055c2e-5ac0-4638-851f-f1051b61d0c2 (platform)
zonegroup 4134640c-d16b-4166-bbd6-987637da469d (prd)
zone 6328c6d7-31a5-4d42-8359-1e28689572da (zone2)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
data sync source: 8adfe5fc-65df-4227-9d85-1d0d1e66ac1f (zone1)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 1 shards
oldest incremental change not applied: 2018-02-03 09:37:03.0.544123s
This is my zone conf:
{
"id": "4134640c-d16b-4166-bbd6-987637da469d",
"name": "platform",
"api_name": "platform",
"is_master": "true",
"endpoints": [
"https://<URL>:443"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "8adfe5fc-65df-4227-9d85-1d0d1e66ac1f",
"zones": [
{
"id": "6328c6d7-31a5-4d42-8359-1e28689572da",
"name": "zone2",
"endpoints": [
"https://<URL>:443"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
},
{
"id": "8adfe5fc-65df-4227-9d85-1d0d1e66ac1f",
"name": "zone2",
"endpoints": [
"https://<URL>:443"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "c6055c2e-5ac0-4638-851f-f1051b61d0c2"
}
Could someone put some light what can be wrong here? Based on the status information is pretty hard to maintain this env. I have to counts the files on both sites to make sure that everything is ok because I can't belive the status information.
And another thing - what is the best way to solve it ? Should I execute sync init --bucket=<bucket name> ?
Actions