Ceph » rbd

Hi Ilya,

Indeed, thanks, it works!
Perhaps, this could be added in the documentation?

Hi,

From my tests, site-name is not sufficient to make snapshot based mirroring work out of the box :

site1=site1
site2=site2
pool=rbd
image=...

rbd mirror pool enable $pool image --site-name $site1 --cluster=$site1
rbd mirror pool enable $pool image --site-name $site2 --cluster=$site2
rbd mirror pool peer add $pool client.rbd-mirror.$site2@$site2 --cluster=$site1
rbd mirror pool peer add $pool client.rbd-mirror.$site1@$site1 --cluster=$site2

rbd mirror pool info $pool --cluster=$site1
# Mode: image
# Site Name: $site1
# 
# Peer Sites:
# 
# UUID: 280eead3-b5ad-4176-aab3-f26f87ec6688
# Name: $site2
# Mirror UUID:
# Direction: rx-tx
# Client: client.rbd-mirror.$site2
# 
# UUID: bdd64a16-a17d-4a28-93b9-c9b4b6cc915f
# Name: 0720436a-37f6-4917-9b47-e013bfd6b14c
# Mirror UUID: 0556bbac-5ba1-4c51-a7b7-fb1c14567ddb
# Direction: tx-only

rbd mirror pool info $pool --cluster=$site2
# Mode: image
# Site Name: cb
# 
# Peer Sites:
# 
# UUID: 16ebbe4c-5128-4222-b2ea-2e6905ca3ec5
# Name: ca
# Mirror UUID:
# Direction: rx-tx
# Client: client.rbd-mirror.ca
# 
# UUID: 63e2606d-aad8-43cd-a600-34f454b174e4
# Name: 26127ba9-b03b-46fe-a809-951929bbd2fc
# Mirror UUID: 32181067-bb4a-4735-9446-3acc7187a45c
# Direction: tx-only

rbd mirror pool status $pool --cluster=$site1 # or $site2
# health: OK
# daemon health: OK
# image health: OK
# images: 0 total

rbd mirror image enable $pool/$image snapshot --cluster=$site1

tail /var/log/ceph/ceph-client.rbd-mirror.$site2.log # on $site2
# 7fa787ebf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5603603a6800 get_replay_status: replay not running

I made a script to be executed on each cluster :

rbd mirror pool info $pool | awk 'BEGIN {FS = ": "} /^UUID/{u=v;v=$2} /^Name/{x=y;y=$2} /^Mirror/{m=n;n=$2}  END{ if (m=="") print "UUID="u"\nUUID2="v"\nNAME2="y"\n"; else print "UUID="v"\nUUID2="u"\nNAME2="x"\n" }'
# UUID=xxx  # peer UUID without Mirror UUID
# UUID2=xxx # peer UUID with Mirror UUID
# NAME2=xxx # peer Name with Mirror UUID

rbd mirror pool peer set $pool $UUID direction tx-only
rbd mirror pool peer set $pool $UUID2 direction rx-tx
rbd mirror pool peer set $pool $UUID site-name zzz
rbd mirror pool peer set $pool $UUID2 site-name $OTHERCLUSTER
rbd mirror pool peer set $pool $UUID site-name $NAME2
rbd mirror pool peer set $pool $UUID2 client client.rbd-mirror.$OTHERCLUSTER

Forgot Ceph version : ceph version 16.2.5 (9b9dd76e12f1907fe5dcc0c1fadadbb784022a42) pacific (stable)

Hmm I no longer have my test clusters but on my end it was working without the trick you did. Could you try using the same client name in both of your clusters (for example client.rbd-mirror-peer like in the documentation)?

Maybe a regression then ? Client name doesn't change anything :

site1=site1
site2=site2
pool=rbd
image=...

for site in $site1 $site2; do ssh $site " \
    ceph auth get-or-create client.rbd-mirror mon 'profile rbd' osd 'profile rbd' -o /etc/ceph/ceph.client.rbd-mirror.keyring \
"; done

scp $site1:/etc/ceph/ceph.client.rbd-mirror.keyring $site2:/etc/ceph/$site1.client.rbd-mirror.keyring
scp $site2:/etc/ceph/ceph.client.rbd-mirror.keyring $site1:/etc/ceph/$site2.client.rbd-mirror.keyring

for site in $site1 $site2; do ssh $site " \
    apt install rbd-mirror; \
    systemctl enable ceph-rbd-mirror.target; \
    systemctl enable --now ceph-rbd-mirror@rbd-mirror.service \
    rbd mirror pool enable $pool image --site-name $site
"; done

rbd mirror pool peer add $pool client.rbd-mirror@$site2 --cluster=$site1
rbd mirror pool peer add $pool client.rbd-mirror@$site1 --cluster=$site2

rbd mirror image enable $pool/$image snapshot --cluster=$site1

ssh $site2 "tail /var/log/ceph/ceph-client.rbd-mirror.log" 
# 7fa7cff03700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55de9e7fc800 init: failed to retrieve mirror peer uuid from remote pool
# 7fa787ebf700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5603603a6800 get_replay_status: replay not running

My workaround came from this thread where direction change seems is enough, but no as there's no client too, and the site-name is incorrect.

In my setup I use a slightly different cephx caps:

ceph auth get-or-create client.rbd-mirror-peer mon 'profile rbd-mirror-peer' osd 'profile rbd'

I also name my systemd services differently ceph-rbd-mirror@rbd-mirror.${id}.service where id is the hostname in my case. The id should fill the --id from the ExecStart command.

Here is the puppet code that we use to do that:

    -> exec { 'ceph-rbd-mirror-create-key':
        command => "ceph --name client.bootstrap-rbd-mirror \
--keyring /var/lib/ceph/bootstrap-rbd-mirror/ceph.keyring   \
auth get-or-create client.rbd-mirror.${id}                  \
mon 'profile rbd-mirror'                                    \
osd 'profile rbd'                                           \
-o /etc/ceph/ceph.client.rbd-mirror.${id}.keyring",
        creates => "/etc/ceph/ceph.client.rbd-mirror.${id}.keyring",
    }
    -> file {"/etc/ceph/ceph.client.rbd-mirror.${id}.keyring":
        owner => 'ceph',
        group => 'ceph',
    }
    -> service { "ceph-rbd-mirror@rbd-mirror.${id}":
        ensure => running,
        enable => true,
    }

By the way is your issue specific to snapshot replication or is it also not working with journal?

Hi Lubo,

Why are you attempting to add a peer manually with "rbd mirror pool peer add"? If you have two clusters, there should be a single "rx-tx" peer in "rbd mirror pool info" output.

I would start from scratch (i.e. remove all peers and disable mirroring on that pool) and use "rbd mirror pool peer bootstrap create" and "rbd mirror pool peer bootstrap import" as described in https://docs.ceph.com/en/latest/rbd/rbd-mirroring/. Snapshot based mirroring definitely does work out of the box, none of the fairly advanced commands you discovered should be needed.

Hi Arthur and Ilya,

Indeed changing rbd-mirror user mon profile from "rbd" to "rbd-mirror-peer" seems to be the problem, now there's only one rx-tx peer per site and it works, meaning Ceph Doc should be updated :

ceph auth get-or-create client.rbd-mirror-peer mon 'profile rbd' osd 'profile rbd'

This issue is specific to snapshot replication, journaling works fine.

I tried to use bootstrap but it doesn't work with ProxMox probably as all conf is in /etc/pve/priv and not /etc/ceph :

rbd mirror pool peer bootstrap create --site-name $site1 $pool --cluster $site1 > token
rbd mirror pool peer bootstrap import --site-name $site2 $pool token --cluster $site2
# 7f0884b721c0 -1 auth: unable to find a keyring on /etc/ceph/..keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory

This warning is harmless, "rbd mirror pool peer bootstrap import" doesn't actually need that keyring.

Hi Ilyua,

Ok, but I can't find any keyring file anywhere. Isn't bootstrap supposed to create it ? rbd-mirror after bootstrap complains about missing key :

7fcaf32db500 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.rbd-mirror.keyring: (2) No such file or directory
7fcaf32db500 -1 AuthRegistry(0x562e6abfaa40) no keyring found at /etc/pve/priv/ceph.client.rbd-mirror.keyring, disabling cephx

No, "rbd mirror pool peer bootstrap import" adds the remote cluster details and key to the config store on the local monitors. It is exactly the point of bootstrap: instead of manually creating the user and making sure that remote cluster ceph.conf and keyring files are copied and named appropriately, you just feed it the base64-encoded token.

Marking this as Resolved because the documentation fixup is now merged but feel free to continue commenting here. If something else needs to be patched, I'll create a new ticket.

Indeed bootstrap does somethings, but documentation doesn't say what.
I compared omap values and after bootstrap rbd_mirroring gets a new pair, is that all ?

mirror_peer_20dacea4-b295-41fa-97b0-e89ba8c388c0
value (101 bytes) :
00000000  02 01 5f 00 00 00 24 00  00 00 32 30 64 61 63 65  |.._...$...20dace|
00000010  61 34 2d 62 32 39 35 2d  34 31 66 61 2d 39 37 62  |a4-b295-41fa-97b|
00000020  30 2d 65 38 39 62 61 38  63 33 38 38 63 30 04 00  |0-e89ba8c388c0..|
00000030  00 00 70 76 61 31 16 00  00 00 63 6c 69 65 6e 74  |..pva1....client|
00000040  2e 72 62 64 2d 6d 69 72  72 6f 72 2d 70 65 65 72  |.rbd-mirror-peer|
00000050  ff ff ff ff ff ff ff ff  02 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00                                    |.....|
00000065

I tried to create a service using the client name client.rbd-mirror-peer :

systemctl enable --now ceph-rbd-mirror@rbd-mirror-peer.service

This fails looking for a keyring :

7f8a8527f500 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.rbd-mirror-peer.keyring: (2) No such file or directory
7f8a8527f500 -1 AuthRegistry(0x555b239caa40) no keyring found at /etc/pve/priv/ceph.client.rbd-mirror-peer.keyring, disabling cephx
7f8a8527f500 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.rbd-mirror-peer.keyring: (2) No such file or directory
7f8a8527f500 -1 AuthRegistry(0x7ffd771e6d20) no keyring found at /etc/pve/priv/ceph.client.rbd-mirror-peer.keyring, disabling cephx
pvb1 rbd-mirror[21981]: failed to fetch mon config (--no-mon-config to skip)

Indeed bootstrap does somethings, but documentation doesn't say what.
I compared omap values and after bootstrap rbd_mirroring gets a new pair, is that all ?

It also adds a couple of entries to the monitor config store. If you compare "Bootstrap Peers" section (which is just "rbd mirror pool peer bootstrap create" + "rbd mirror pool peer bootstrap import") with "Add Cluster Peer Manually" section you would see that bootstrap automates quite a lot: creating a cephx user and copying its key over to the remote cluster, mirror peer setup and remote cluster access setup on each site. Bootstrap eliminates the need for placing and maintaining appropriately named ceph.conf and keyring files for each remote cluster in the right location.

The spurious warnings are being addressed in https://tracker.ceph.com/issues/51628.