Ceph » rgw

Steps:
------
On an existing Active/Active multisite configuration, added a new node to configure a Active/Passive configuration.
Command which I executed to make the rgw node readonly :-
$ radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-3 --access-key=${access_key} --secret=${secret} --endpoints=http://rgw2:80 --default --readonly
let me know if this is the right command to make the node read-only

Output of radosgw-admin period get --staging :- http://pastebin.com/AQKndvyG , us-3 zone has been set to readonly

Once the zone add was completed, restarted rgw services on both the nodes.
Only the metadata was seen in the pools and actual container and buckets were missing.
[root@rgw3 ~]# rados ls -p us-3.rgw.data.root
.bucket.meta.Container1:ee31c3bd-8907-407c-a58c-30aa23c2be62.54105.2
.bucket.meta.Container2:ee31c3bd-8907-407c-a58c-30aa23c2be62.54149.1
.bucket.meta.my-new-bucket:ee31c3bd-8907-407c-a58c-30aa23c2be62.54105.1

When checked the sync status it failed with permission denied error.
Restarted the rgw service again but no luck.

[root@rgw1 ceph]# radosgw-admin sync status
realm 19cf8f2b-2368-486c-9592-7225cff4bf13 (earth)
zonegroup 8e358edb-cd6e-4cf9-91bc-59df68e8a86f (us)
zone ee31c3bd-8907-407c-a58c-30aa23c2be62 (us-1)
metadata sync no sync (zone is master)
2016-04-20 09:23:09.313212 7f1ebb861a40 0 ERROR: failed to fetch datalog info
2016-04-20 09:23:09.317088 7f1ebb861a40 0 ERROR: failed to fetch datalog info
data sync source: 22ce2326-a251-49b1-8ddb-c73428156b48 (us-3)
failed to retrieve sync info: (5) Input/output error
source: ea9e8433-4e45-4251-ac7e-4298af61653c (us-2)
failed to retrieve sync info: (13) Permission denied

[root@rgw3 ~]# radosgw-admin sync status
realm 19cf8f2b-2368-486c-9592-7225cff4bf13 (earth)
zonegroup 8e358edb-cd6e-4cf9-91bc-59df68e8a86f (us)
zone 22ce2326-a251-49b1-8ddb-c73428156b48 (us-3)
metadata sync failed to read sync status: (2) No such file or directory
2016-04-20 09:23:50.242114 7f9f28396a40 0 ERROR: failed to fetch datalog info
data sync source: ea9e8433-4e45-4251-ac7e-4298af61653c (us-2)
failed to retrieve sync info: (13) Permission denied
source: ee31c3bd-8907-407c-a58c-30aa23c2be62 (us-1)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source
Ceph Version :- v10.1.1

Additional Info :-
Note that rgw1 and rgw2 nodes were already configured with active/active site, now rgw3 is the new addition and it has been set to readonly.
SO that rgw1 and rgw3 can act as active/passive DR setup

Master zone log :- rgw1
----------------------
2016-04-20 09:30:25.316434 7ff8777fe700 0 ERROR: failed to fetch datalog info
2016-04-20 09:30:45.311773 7ff8817fa700 0 ERROR: failed to fetch datalog info
2016-04-20 09:30:45.316541 7ff8777fe700 0 ERROR: failed to fetch datalog info
2016-04-20 09:30:47.525290 7ff7a4fc9700 0 ERROR: failed to wait for op, ret=-11: POST http://rgw3:8080/admin/realm/period?period=5d47c9e2-c49f-4f12-b4a5-5986069ebcc0&epoch=3&rgwx-zonegr
oup=8e358edb-cd6e-4cf9-91bc-59df68e8a86f
2016-04-20 09:30:50.530628 7ff7a4fc9700 0 ERROR: failed to wait for op, ret=-13: POST http://rgw2:8080/admin/realm/period?period=5d47c9e2-c49f-4f12-b4a5-5986069ebcc0&epoch=3&rgwx-zonegr
oup=8e358edb-cd6e-4cf9-91bc-59df68e8a86f

http://pastebin.com/B4pCN8Dp

On Read-only node:- rgw3
------------------------
Logs :- http://pastebin.com/8PdJMh95