Bug #50970
closedreplication: local site naming in manual mode and snapshot mirroring
Added by Arthur Outhenin-Chalandre almost 3 years ago. Updated over 2 years ago.
0%
Description
From my testing, snapshot mirroring is more strict on the coherence of the site name: local site name and remote site name in the remote site should be equal. This leads to error in the init of the replayer (this led to this issue https://tracker.ceph.com/issues/50931 for the record). While with non coherent name, journaling is working fine.
This is fine to be strict on the site name but it's theorically (at least to my knowledge) not possible to change the local site name with the manual setup (https://docs.ceph.com/en/latest/rbd/rbd-mirroring/#add-cluster-peer-manually). I have a hacky workaround for this which is to create a bootstrap token even if you have done the manual setup.
In my case this ends up in error (with the "manual mode" already setup):
```
$ rbd mirror pool peer bootstrap create --site-name my_local_site volumes
rbd: mismatch with pre-existing RBD mirroring peer user caps
```
But it's essentially fine, the local site name change is done before this error :).
So maybe a command to change the local site name in the rbd cli would be handy?
Updated by Ilya Dryomov almost 3 years ago
- Assignee set to Ilya Dryomov
Hi Arthur,
You can set the site name when you enable mirroring on the pool:
$ rbd mirror pool enable --site-name site1 rbd image $ rbd mirror pool info Mode: image Site Name: site1 Peer Sites: none
And you should be able to change it later using the same command:
$ rbd mirror pool enable --site-name site2 rbd image $ rbd mirror pool info Mode: image Site Name: site2 Peer Sites: none
Updated by Ilya Dryomov almost 3 years ago
- Status changed from New to Need More Info
Updated by Arthur Outhenin-Chalandre almost 3 years ago
Hi Ilya,
Indeed, thanks, it works!
Perhaps, this could be added in the documentation?
Updated by Ilya Dryomov almost 3 years ago
- Status changed from Need More Info to In Progress
Yeah, I'll treat this ticket as a doc enhancement request then.
Updated by Lubo Fr over 2 years ago
Hi,
From my tests, site-name is not sufficient to make snapshot based mirroring work out of the box :
site1=site1 site2=site2 pool=rbd image=... rbd mirror pool enable $pool image --site-name $site1 --cluster=$site1 rbd mirror pool enable $pool image --site-name $site2 --cluster=$site2 rbd mirror pool peer add $pool client.rbd-mirror.$site2@$site2 --cluster=$site1 rbd mirror pool peer add $pool client.rbd-mirror.$site1@$site1 --cluster=$site2 rbd mirror pool info $pool --cluster=$site1 # Mode: image # Site Name: $site1 # # Peer Sites: # # UUID: 280eead3-b5ad-4176-aab3-f26f87ec6688 # Name: $site2 # Mirror UUID: # Direction: rx-tx # Client: client.rbd-mirror.$site2 # # UUID: bdd64a16-a17d-4a28-93b9-c9b4b6cc915f # Name: 0720436a-37f6-4917-9b47-e013bfd6b14c # Mirror UUID: 0556bbac-5ba1-4c51-a7b7-fb1c14567ddb # Direction: tx-only rbd mirror pool info $pool --cluster=$site2 # Mode: image # Site Name: cb # # Peer Sites: # # UUID: 16ebbe4c-5128-4222-b2ea-2e6905ca3ec5 # Name: ca # Mirror UUID: # Direction: rx-tx # Client: client.rbd-mirror.ca # # UUID: 63e2606d-aad8-43cd-a600-34f454b174e4 # Name: 26127ba9-b03b-46fe-a809-951929bbd2fc # Mirror UUID: 32181067-bb4a-4735-9446-3acc7187a45c # Direction: tx-only rbd mirror pool status $pool --cluster=$site1 # or $site2 # health: OK # daemon health: OK # image health: OK # images: 0 total rbd mirror image enable $pool/$image snapshot --cluster=$site1 tail /var/log/ceph/ceph-client.rbd-mirror.$site2.log # on $site2 # 7fa787ebf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5603603a6800 get_replay_status: replay not runningThe only way I found to manage is to exchange peer data so that the peer with Mirror UUID becomes rx-tx :
| UUID | $UUID | |
| Name | $CLUSTER | $NAME2 |
| Mirror UUID | ||
| Direction | rx-tx | tx-only |
| Client | client.rbd-mirror.$CLUSTER | |
| UUID | $UUID2 | |
| Name | $NAME2 | $CLUSTER |
| Mirror UUID | xxxxxxxxxx | |
| Direction | tx-only | rx-tx |
| Client | client.rbd-mirror.$CLUSTER | |
I made a script to be executed on each cluster :
rbd mirror pool info $pool | awk 'BEGIN {FS = ": "} /^UUID/{u=v;v=$2} /^Name/{x=y;y=$2} /^Mirror/{m=n;n=$2} END{ if (m=="") print "UUID="u"\nUUID2="v"\nNAME2="y"\n"; else print "UUID="v"\nUUID2="u"\nNAME2="x"\n" }'
# UUID=xxx # peer UUID without Mirror UUID
# UUID2=xxx # peer UUID with Mirror UUID
# NAME2=xxx # peer Name with Mirror UUID
rbd mirror pool peer set $pool $UUID direction tx-only
rbd mirror pool peer set $pool $UUID2 direction rx-tx
rbd mirror pool peer set $pool $UUID site-name zzz
rbd mirror pool peer set $pool $UUID2 site-name $OTHERCLUSTER
rbd mirror pool peer set $pool $UUID site-name $NAME2
rbd mirror pool peer set $pool $UUID2 client client.rbd-mirror.$OTHERCLUSTER
Updated by Lubo Fr over 2 years ago
Forgot Ceph version : ceph version 16.2.5 (9b9dd76e12f1907fe5dcc0c1fadadbb784022a42) pacific (stable)
Updated by Arthur Outhenin-Chalandre over 2 years ago
Hmm I no longer have my test clusters but on my end it was working without the trick you did. Could you try using the same client name in both of your clusters (for example client.rbd-mirror-peer like in the documentation)?
Updated by Lubo Fr over 2 years ago
Maybe a regression then ? Client name doesn't change anything :
site1=site1
site2=site2
pool=rbd
image=...
for site in $site1 $site2; do ssh $site " \
ceph auth get-or-create client.rbd-mirror mon 'profile rbd' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirror.keyring \
"; done
scp $site1:/etc/ceph/ceph.client.rbd-mirror.keyring $site2:/etc/ceph/$site1.client.rbd-mirror.keyring
scp $site2:/etc/ceph/ceph.client.rbd-mirror.keyring $site1:/etc/ceph/$site2.client.rbd-mirror.keyring
for site in $site1 $site2; do ssh $site " \
apt install rbd-mirror; \
systemctl enable ceph-rbd-mirror.target; \
systemctl enable --now ceph-rbd-mirror@rbd-mirror.service \
rbd mirror pool enable $pool image --site-name $site
"; done
rbd mirror pool peer add $pool client.rbd-mirror@$site2 --cluster=$site1
rbd mirror pool peer add $pool client.rbd-mirror@$site1 --cluster=$site2
rbd mirror image enable $pool/$image snapshot --cluster=$site1
ssh $site2 "tail /var/log/ceph/ceph-client.rbd-mirror.log"
# 7fa7cff03700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55de9e7fc800 init: failed to retrieve mirror peer uuid from remote pool
# 7fa787ebf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5603603a6800 get_replay_status: replay not running
My workaround came from this thread where direction change seems is enough, but no as there's no client too, and the site-name is incorrect.
Updated by Arthur Outhenin-Chalandre over 2 years ago
In my setup I use a slightly different cephx caps:
ceph auth get-or-create client.rbd-mirror-peer mon 'profile rbd-mirror-peer' osd 'profile rbd'
I also name my systemd services differently ceph-rbd-mirror@rbd-mirror.${id}.service where id is the hostname in my case. The id should fill the --id from the ExecStart command.
Here is the puppet code that we use to do that:
-> exec { 'ceph-rbd-mirror-create-key':
command => "ceph --name client.bootstrap-rbd-mirror \
--keyring /var/lib/ceph/bootstrap-rbd-mirror/ceph.keyring \
auth get-or-create client.rbd-mirror.${id} \
mon 'profile rbd-mirror' \
osd 'profile rbd' \
-o /etc/ceph/ceph.client.rbd-mirror.${id}.keyring",
creates => "/etc/ceph/ceph.client.rbd-mirror.${id}.keyring",
}
-> file {"/etc/ceph/ceph.client.rbd-mirror.${id}.keyring":
owner => 'ceph',
group => 'ceph',
}
-> service { "ceph-rbd-mirror@rbd-mirror.${id}":
ensure => running,
enable => true,
}
By the way is your issue specific to snapshot replication or is it also not working with journal?
Updated by Ilya Dryomov over 2 years ago
Hi Lubo,
Why are you attempting to add a peer manually with "rbd mirror pool peer add"? If you have two clusters, there should be a single "rx-tx" peer in "rbd mirror pool info" output.
I would start from scratch (i.e. remove all peers and disable mirroring on that pool) and use "rbd mirror pool peer bootstrap create" and "rbd mirror pool peer bootstrap import" as described in https://docs.ceph.com/en/latest/rbd/rbd-mirroring/. Snapshot based mirroring definitely does work out of the box, none of the fairly advanced commands you discovered should be needed.
Updated by Lubo Fr over 2 years ago
Hi Arthur and Ilya,
Indeed changing rbd-mirror user mon profile from "rbd" to "rbd-mirror-peer" seems to be the problem, now there's only one rx-tx peer per site and it works, meaning Ceph Doc should be updated :
ceph auth get-or-create client.rbd-mirror-peer mon 'profile rbd' osd 'profile rbd'.
This issue is specific to snapshot replication, journaling works fine.
I tried to use bootstrap but it doesn't work with ProxMox probably as all conf is in /etc/pve/priv and not /etc/ceph :
rbd mirror pool peer bootstrap create --site-name $site1 $pool --cluster $site1 > token rbd mirror pool peer bootstrap import --site-name $site2 $pool token --cluster $site2 # 7f0884b721c0 -1 auth: unable to find a keyring on /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
Updated by Ilya Dryomov over 2 years ago
This warning is harmless, "rbd mirror pool peer bootstrap import" doesn't actually need that keyring.
Updated by Ilya Dryomov over 2 years ago
- Status changed from In Progress to Fix Under Review
- Pull request ID set to 43047
Updated by Lubo Fr over 2 years ago
Hi Ilyua,
Ok, but I can't find any keyring file anywhere. Isn't bootstrap supposed to create it ? rbd-mirror after bootstrap complains about missing key :
7fcaf32db500 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.rbd-mirror.keyring: (2) No such file or directory 7fcaf32db500 -1 AuthRegistry(0x562e6abfaa40) no keyring found at /etc/pve/priv/ceph.client.rbd-mirror.keyring, disabling cephx
Updated by Ilya Dryomov over 2 years ago
No, "rbd mirror pool peer bootstrap import" adds the remote cluster details and key to the config store on the local monitors. It is exactly the point of bootstrap: instead of manually creating the user and making sure that remote cluster ceph.conf and keyring files are copied and named appropriately, you just feed it the base64-encoded token.
Updated by Ilya Dryomov over 2 years ago
Marking this as Resolved because the documentation fixup is now merged but feel free to continue commenting here. If something else needs to be patched, I'll create a new ticket.
Updated by Ilya Dryomov over 2 years ago
- Status changed from Fix Under Review to Resolved
Updated by Lubo Fr over 2 years ago
Indeed bootstrap does somethings, but documentation doesn't say what.
I compared omap values and after bootstrap rbd_mirroring gets a new pair, is that all ?
mirror_peer_20dacea4-b295-41fa-97b0-e89ba8c388c0 value (101 bytes) : 00000000 02 01 5f 00 00 00 24 00 00 00 32 30 64 61 63 65 |.._...$...20dace| 00000010 61 34 2d 62 32 39 35 2d 34 31 66 61 2d 39 37 62 |a4-b295-41fa-97b| 00000020 30 2d 65 38 39 62 61 38 63 33 38 38 63 30 04 00 |0-e89ba8c388c0..| 00000030 00 00 70 76 61 31 16 00 00 00 63 6c 69 65 6e 74 |..pva1....client| 00000040 2e 72 62 64 2d 6d 69 72 72 6f 72 2d 70 65 65 72 |.rbd-mirror-peer| 00000050 ff ff ff ff ff ff ff ff 02 00 00 00 00 00 00 00 |................| 00000060 00 00 00 00 00 |.....| 00000065
I tried to create a service using the client name client.rbd-mirror-peer :
systemctl enable --now ceph-rbd-mirror@rbd-mirror-peer.service
This fails looking for a keyring :
7f8a8527f500 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.rbd-mirror-peer.keyring: (2) No such file or directory 7f8a8527f500 -1 AuthRegistry(0x555b239caa40) no keyring found at /etc/pve/priv/ceph.client.rbd-mirror-peer.keyring, disabling cephx 7f8a8527f500 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.rbd-mirror-peer.keyring: (2) No such file or directory 7f8a8527f500 -1 AuthRegistry(0x7ffd771e6d20) no keyring found at /etc/pve/priv/ceph.client.rbd-mirror-peer.keyring, disabling cephx pvb1 rbd-mirror[21981]: failed to fetch mon config (--no-mon-config to skip)
Updated by Ilya Dryomov over 2 years ago
Indeed bootstrap does somethings, but documentation doesn't say what.
I compared omap values and after bootstrap rbd_mirroring gets a new pair, is that all ?
It also adds a couple of entries to the monitor config store. If you compare "Bootstrap Peers" section (which is just "rbd mirror pool peer bootstrap create" + "rbd mirror pool peer bootstrap import") with "Add Cluster Peer Manually" section you would see that bootstrap automates quite a lot: creating a cephx user and copying its key over to the remote cluster, mirror peer setup and remote cluster access setup on each site. Bootstrap eliminates the need for placing and maintaining appropriately named ceph.conf and keyring files for each remote cluster in the right location.
The spurious warnings are being addressed in https://tracker.ceph.com/issues/51628.