Ceph » rbd

Hello Team,
Please provide any solution for that issue we are stuck with 200TIB data at the dc and 136TIB data only at dr site. now we are in the middle of the situation , kindly suggest any solution .

rbd_mirror_journal_max_fetch_bytes option was removed in the changeset that you linked above -- https://github.com/ceph/ceph/pull/27670. I see the two images that are millions of journal entries behind though. Perhaps Mykola has a suggestion?

Ilya Dryomov wrote:

rbd_mirror_journal_max_fetch_bytes option was removed in the changeset that you linked above -- https://github.com/ceph/ceph/pull/27670.

It's still mentioned in one place in the documentation, by accident. I'm cleaning that up in https://github.com/ceph/ceph/pull/51964.

Ilya Dryomov wrote:

Perhaps Mykola has a suggestion?

Not much. I assume here that replaying is just slow and can't catch with primary image updates? I.e. it is not a situation when it gets stuck for some images (in this case I suppose you should observe images with large "entries behind primary" and 0 "bytes per second" on dashboard mirror page. Still it might be useful to attach `rbd mirror pool $pool --verbose` output collected on both sites, as it contains more detailed information.

If it is just slow, then max_fetch_bytes is indeed the first suspect. Previously it was configurable but with small default, and increasing it speeded up replaying. Now it is calculated based on available cache (controlled by rbd_mirror_memory_target) and the number of images to replay. It would be good to check what value you have. The only way I know is to set `debug_journaler=10` for a couple of minutes and then look for entries like this in the rbd-mirror logs:

2023-06-12T13:30:17.724+0100 7fb0cb341840 10 JournalPlayer: 0x5592c934bb80 handle_cache_rebalanced: new_cache_bytes=134217728, max_fetch_bytes=33554432

If you see the number more than 16Mb (preferable 32M) then you are ok. If not, then try to increase rbd_mirror_memory_target.

If the max_fetch value is large and it is still slow, then I would look in the log how much time it spends on fetching an object and processing it, i.e. looking at lines like these:

2023-06-12T13:29:23.729+0100 7fb0c42db6c0 10 ObjectPlayer: 0x5592ceee61c0 fetch: journal_data.4.114c69f46ce5.2
2023-06-12T13:29:23.729+0100 7fb0ba2c76c0 10 ObjectPlayer: 0x5592ceee61c0 handle_fetch_complete: journal_data.4.114c69f46ce5.2, r=0, len=0

and calculating time difference between "fetch" and "handle_fetch_complete" for a particular object. (In this example the object was journal_data.4.114c69f46ce5.2 and the time difference is 0, but in your case I expect it will not be 0). If you have problems to interpret the log just attach it here. Then I suggest increase the debug_journaler to 30 to have all debug messages.

The trickiest part thought will be to enable debug and set log_to_file config variables in a conterized environment. It is easy for osd daemons, as the mgr updates them on on the fly when sees changes in `ceph config set osd ...`. But for rbd mirror I think it will not work. The only way I can think of is to use the rbd mirror admin socket. I.e. on the secondary site (where replaying rbd-mirrors are running), enter the rbd-mirror container:

cephadm enter --name rbd-mirror.XYZ

list all admin sockets:

ls /var/run/ceph/client.mirror.*.asok

and for every socket run:

ceph daemon /var/run/ceph/client.mirror.XYZ.asok config set log_to_file true
  ceph daemon /var/run/ceph/client.mirror.XYZ.asok config set debug_journaler 30

To disable (after running for several minutes):

ceph daemon /var/run/ceph/client.mirror.XYZ.asok config set debug_journaler 1/5
  ceph daemon /var/run/ceph/client.mirror.XYZ.asok config set log_to_file false

You should find the logs on the host in /var/log/ceph/{fsid} directory with client.mirror.*log names. Look for logs with large size and that have JournalPlayer and ObjectPlayer entries.

Note currently I don't have a cephadm based environment with rbd-mirror deployed where I could check my instructions, so there might be a mistake above, still hope it will work for you or you will be able to figure out how to adjust it for your environment.

May be Ilya will suggest an easier way to turn on/off debug.

Hello Mykola Golub,
Thanks for your responce , we have also try to change the value of the rbd_mirror_memory_target , but we don't see any impact on our mirroring b/w clusters . But one thing we notice when we are using the snapshot based mirroing we are geeting proper speed in the mirroring . we are faceing only issue in the journal based rbd-mirroring. if you or community have ayn other input that may help full to us.