Project

General

Profile

Actions
Copy link

Bug #21329

closed

rados gateway multi-site failed to sync

Added by hoan nv over 6 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
Ceph - v12.2.0
ceph-qa-suite:
rgw
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi all.

I config multi-site rados gateway use this guide.

http://docs.ceph.com/docs/master/radosgw/multisite/

After config, status of master node.

root@hoannv-ceph-luminous-1:/etc/ceph# radosgw-admin sync status
realm 07966f39-05f8-4da3-820e-15864dda3ca9 (movies)
zonegroup db34009a-7776-46ae-b94f-e81455254df3 (vczone)
zone f5e02a6f-be13-414d-914f-8690daad5710 (vczone-nps)
metadata sync no sync (zone is master)
2017-09-09 09:44:18.498364 7f08dda9dcc0 0 data sync: ERROR: failed to fetch datalog info
data sync source: d5cef7a6-c5a1-4f12-af87-6bf598fd2491 (vczone-ntl)
failed to retrieve sync info: (13) Permission denied

secondary node sync status

root@hoannv-ceph-luminous-1-zone-2:/etc/ceph# radosgw-admin sync status
realm 07966f39-05f8-4da3-820e-15864dda3ca9 (movies)
zonegroup db34009a-7776-46ae-b94f-e81455254df3 (vczone)
zone d5cef7a6-c5a1-4f12-af87-6bf598fd2491 (vczone-ntl)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
data sync source: f5e02a6f-be13-414d-914f-8690daad5710 (vczone-nps)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source

This is all my command.

--- master zone
radosgw-admin realm create --rgw-realm=movies --default
radosgw-admin zonegroup create --rgw-zonegroup=vczone --endpoints=http://10.5.8.88:80 --rgw-realm=movies --master --default
radosgw-admin zone create --rgw-zonegroup=vczone --rgw-zone=vczone-nps --endpoints=http://10.5.8.88:80

radosgw-admin zonegroup remove --rgw-zonegroup=default --rgw-zone=default
radosgw-admin period update --commit
radosgw-admin zone delete --rgw-zone=default
radosgw-admin period update --commit
radosgw-admin zonegroup delete --rgw-zonegroup=default
radosgw-admin period update --commit

for i in default.rgw.control default.rgw.meta default.rgw.log;do rados rmpool ${i} ${i} --yes-i-really-really-mean-it;done

radosgw-admin user create --uid="synchronization-user" --display-name="Synchronization User" --system

radosgw-admin period update --commit

systemctl restart ceph-radosgw@rgw.`hostname -s`

--- secondary zone
radosgw-admin realm pull --url=http://10.5.8.88:80 --access-key=26P94IXSBGX49BH9VU3X --secret=2UtPPWpzBexNf4SicMjyf2phNjQvEidH5JefQAWC
radosgw-admin period pull --url=http://10.5.8.88:80 --access-key=26P94IXSBGX49BH9VU3X --secret=2UtPPWpzBexNf4SicMjyf2phNjQvEidH5JefQAWC
radosgw-admin zone create --rgw-zonegroup=vczone --rgw-zone=vczone-ntl --endpoints=http://10.5.8.74:80 --access-key=26P94IXSBGX49BH9VU3X --secret=2UtPPWpzBexNf4SicMjyf2phNjQvEidH5JefQAWC

for i in default.rgw.control default.rgw.meta default.rgw.log;do rados rmpool ${i} ${i} --yes-i-really-really-mean-it;done

radosgw-admin period update --commit

systemctl restart ceph-radosgw@rgw.`hostname -s`

Log in attachment file.

Thanks.

Files

ceph-client.rgw.zip (574 KB) ceph-client.rgw.zip hoan nv, 09/09/2017 02:56 AM
Actions
Copy link
 #1

Updated by Nathan Cutler over 6 years ago

  • Project changed from Ceph to rgw
Actions
Copy link
 #2

Updated by hoan nv over 6 years ago

log master zone

> /var/log/ceph/ceph-client.rgw.hoannv-ceph-luminous-1.log <
2017-09-13 10:34:00.502941 7fdd72b22700 1 ====== starting new request req=0x7fdd72b1c1f0 =====
2017-09-13 10:34:00.505116 7fdd72b22700 1 ====== req done req=0x7fdd72b1c1f0 op status=0 http_status=200 ======
2017-09-13 10:34:00.505172 7fdd72b22700 1 civetweb: 0x6d8c1c6000: 10.5.8.74 - - [13/Sep/2017:06:28:33 +0700] "GET /admin/log?type=metadata&id=63&period=487a3057-60be-4d61-8720-eb2897900b71&max-entries=100&&rgwx-zonegroup=db34009a-7776-46ae-b94f-e81455254df3 HTTP/1.1" 1 0 - - > /var/log/ceph/ceph-mgr.hoannv-ceph-luminous-1.log <
2017-09-13 10:34:01.048738 7f8c84ff8700 1 mgr send_beacon active > /var/log/ceph/ceph-mon.hoannv-ceph-luminous-1.log <
2017-09-13 10:34:01.536263 7fd9b1adf700 0 mon.hoannv-ceph-luminous-1@1(peon).data_health(34) update_stats avail 91% total 64498 MB, used 2744 MB, avail 59098 MB > /var/log/ceph/ceph-client.rgw.hoannv-ceph-luminous-1.log <
2017-09-13 10:34:02.603720 7fdd9fd83700 0 data sync: ERROR: failed to fetch datalog info
2017-09-13 10:34:02.754777 7fdd72b22700 1 ====== starting new request req=0x7fdd72b1c1f0 =====
2017-09-13 10:34:02.756553 7fdd72b22700 1 ====== req done req=0x7fdd72b1c1f0 op status=0 http_status=200 ======
2017-09-13 10:34:02.756607 7fdd72b22700 1 civetweb: 0x6d8c1c6000: 10.5.8.74 - - [13/Sep/2017:06:28:33 +0700] "GET /admin/log?type=metadata&id=3&period=487a3057-60be-4d61-8720-eb2897900b71&max-entries=100&&
rgwx-zonegroup=db34009a-7776-46ae-b94f-e81455254df3 HTTP/1.1" 1 0 - - > /var/log/ceph/ceph-mgr.hoannv-ceph-luminous-1.log <
2017-09-13 10:34:03.049440 7f8c84ff8700 1 mgr send_beacon active > /var/log/ceph/ceph-client.rgw.hoannv-ceph-luminous-1.log <
2017-09-13 10:34:03.662665 7fdd72b22700 1 ====== starting new request req=0x7fdd72b1c1f0 =====
2017-09-13 10:34:03.664785 7fdd72b22700 1 ====== req done req=0x7fdd72b1c1f0 op status=0 http_status=200 ======
2017-09-13 10:34:03.664832 7fdd72b22700 1 civetweb: 0x6d8c1c6000: 10.5.8.74 - - [13/Sep/2017:06:28:33 +0700] "GET /admin/log?type=metadata&id=9&period=487a3057-60be-4d61-8720-eb2897900b71&max-entries=100&&
rgwx-zonegroup=db34009a-7776-46ae-b94f-e81455254df3 HTTP/1.1" 1 0 - -
2017-09-13 10:34:03.922128 7fdd72b22700 1 ====== starting new request req=0x7fdd72b1c1f0 =====
2017-09-13 10:34:03.924847 7fdd72b22700 1 ====== req done req=0x7fdd72b1c1f0 op status=0 http_status=200 ======
2017-09-13 10:34:03.924894 7fdd72b22700 1 civetweb: 0x6d8c1c6000: 10.5.8.74 - - [13/Sep/2017:06:28:33 +0700] "GET /admin/log?type=metadata&id=2&period=487a3057-60be-4d61-8720-eb2897900b71&max-entries=100&&
rgwx-zonegroup=db34009a-7776-46ae-b94f-e81455254df3 HTTP/1.1" 1 0 - -
2017-09-13 10:34:04.125128 7fdd72b22700 1 ====== starting new request req=0x7fdd72b1c1f0 =====
2017-09-13 10:34:04.127026 7fdd72b22700 1 ====== req done req=0x7fdd72b1c1f0 op status=0 http_status=200 ======
2017-09-13 10:34:04.127069 7fdd72b22700 1 civetweb: 0x6d8c1c6000: 10.5.8.74 - - [13/Sep/2017:06:28:33 +0700] "GET /admin/log?type=metadata&id=39&period=487a3057-60be-4d61-8720-eb2897900b71&max-entries=100&
&rgwx-zonegroup=db34009a-7776-46ae-b94f-e81455254df3 HTTP/1.1" 1 0 - -

log secondary zone

2017-09-13 10:32:17.971232 7f360a19a700 1 meta sync: ERROR: failed to read mdlog info with (2) No such file or directory
2017-09-13 10:32:19.660778 7f360a19a700 1 meta sync: ERROR: failed to read mdlog info with (2) No such file or directory
2017-09-13 10:32:22.603664 7f3514da9700 1 ====== starting new request req=0x7f3514da31f0 =====
2017-09-13 10:32:22.603935 7f3514da9700 1 ====== req done req=0x7f3514da31f0 op status=0 http_status=403 ======
2017-09-13 10:32:22.603988 7f3514da9700 1 civetweb: 0xa58d251000: 10.5.8.88 - - [13/Sep/2017:10:32:22 +0700] "GET /admin/log?type=data&rgwx-zonegroup=db34009a-7776-46ae-b94f-e81455254df3 HTTP/1.1" 1 0 - -
2017-09-13 10:32:22.629490 7f360a19a700 1 meta sync: ERROR: failed to read mdlog info with (2) No such file or directory
2017-09-13 10:32:23.089541 7f360a19a700 1 meta sync: ERROR: failed to read mdlog info with (2) No such file or directory

Actions
Copy link
 #3

Updated by Casey Bodley over 6 years ago

It looks like you're creating the sync user and putting its keys in the secondary zone, but you're not setting the keys in the primary zone. On the primary cluster, try this?

radosgw-admin zone modify --rgw-zone vczone-nps --access-key=26P94IXSBGX49BH9VU3X --secret=2UtPPWpzBexNf4SicMjyf2phNjQvEidH5JefQAWC
radosgw-admin period update --commit

Actions
Copy link
 #4

Updated by Casey Bodley over 6 years ago

  • Status changed from New to Need More Info
Actions
Copy link
 #5

Updated by hoan nv over 6 years ago

Casey Bodley wrote:

It looks like you're creating the sync user and putting its keys in the secondary zone, but you're not setting the keys in the primary zone. On the primary cluster, try this?
[...]

I tried it. It not working.

This access-key and secret-key is default value.
I need change default value?

Thanks.
Hoan

Actions
Copy link
 #6

Updated by hoan nv over 6 years ago

Yes if fixed.

Thanks.

Actions
Copy link
 #7

Updated by hyun ha over 4 years ago

hoan nv wrote:

Yes if fixed.

Thanks.

hi, I have same problem as above issue with ceph rgw version 12.2.11. how did you solved above issue?

Actions
Copy link
 #8

Updated by hoan nv over 4 years ago

hyun ha wrote:

hoan nv wrote:

Yes if fixed.

Thanks.

hi, I have same problem as above issue with ceph rgw version 12.2.11. how did you solved above issue?

I changed zones's access-key and secret-key to key of system user.

Actions
Copy link
 #9

Updated by hyun ha over 4 years ago

hoan nv wrote:

hyun ha wrote:

hoan nv wrote:

Yes if fixed.

Thanks.

hi, I have same problem as above issue with ceph rgw version 12.2.11. how did you solved above issue?

I changed zones's access-key and secret-key to key of system user.

thanks for quick reply!

Actions
Copy link
 #10

Updated by Casey Bodley over 2 years ago

  • Status changed from Need More Info to Resolved
Actions
Copy link

Also available in: Atom PDF