Ceph » CephFS

Mathias Kuhring wrote:

We might have found a major performance bug in the cephfs snapshot mirroring.
We already reported it to the mailing list but with now answers, yet.
So we will mainly reuse the details from here:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/R7NRB5PNGHL4YJ7EZ5TYBLLJFTU7I2MK/

We are using ceph version 17.2.1
(ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable).
We are using cephadm since version 15 octopus.

We mirror several CephFS directories from our main cluster our to a second mirror cluster.
In particular with bigger directories (over 900 TB and 186 M of files), we noticed that mirroring is very slow.
On the mirror, most of the time we only observe a write speed of 0 to 10 MB/s in the client IO.
The target peer directory often doesn't show increase in size during synchronization
(when we check with: getfattr -n ceph.dir.rbytes).

The status of the syncs is always fine, i.e. syncing and not failing, e.g.:

[...]

The cluster nodes (6 per main and mirror cluster, each) are connected with Dual 40G NICs to the switches.
Connection between switches are 2x 100G.
Simple write operations from other clients to the mirror cephfs result in writes of e.g. 300 to 400 MB/s.
So network doesn't seem to be the issue, here.

We started to dig into debug logs of the cephfs-mirror daemon / docker container.
We set the debug level to 20. Otherwise there are no messages at all (so no errors).

We observed a lot of messages with need_data_sync=0, need_attr_sync=1.
Leading us to the assumption, that instead of actual data a lot of attributes are synced.

We started looking at specific examples in the logs and tried to make sense from the source code which steps are happening.
Most of the messages are coming from cephfs::mirror::PeerReplayer
https://github.com/ceph/ceph/blob/6fee777d603aebce492c57b41f3b5760d50ddb07/src/tools/cephfs_mirror/PeerReplayer.cc

We figured, the do_synchronize function checks if data (need_data_sync) or attributes (need_attr_sync) should be synchronized using should_sync_entry.
And if necessary performs the sync using remote_file_op.
should_sync_entry reports different ctimes for our examples, e.g.:
[...]

Based on these different ctimes, should_sync_entry decides then that attributes need to be synced:
*need_attr_sync = (cstx.stx_ctime != pstx.stx_ctime)
https://github.com/ceph/ceph/blob/6fee777d603aebce492c57b41f3b5760d50ddb07/src/tools/cephfs_mirror/PeerReplayer.cc#L911

We assume cur statx/cstx refers to the file in the snapshot currently mirrored.
But what exactly is prev statx/pstx?

So, the mirror daemon can optimize the directory comparison in the following case: Say you have 3 snapshots in the primary site - snap_a, snap_b and snap_c. Assume that the secondary site is empty. Synchronizing snap_a requires a bulk transfer. Now when snap_b is to be synchronized, the mirror daemon would try to infer the delta between the two snapshots by walking the directories of snapshots (one in primary and the other in secondary). The optimization here is to use the "local" (primary) snapshot (snap_a) for attribute comparison rather then the secondary copy (which might be over a WAN, slow link). Therefore, snap_b is compared against "local" snap_a, snap_c against "local" snap_b.

Is it the peer path or the last snapshot on the mirror peer?

Last transferred snapshot, but the one on the primary site. Note that the mirror daemon ensures that it can rely this optimization for correctness, else it may lead to incorrect data getting synchronized.

We can confirm that ctimes are different on the main cluster and the mirror.

They will be different since the mirror daemon transfers file to the secondary cluster and then takes a snapshot.

On the main cluster, the ctimes are consistent in every snapshot (since the files didn't change).

If they are same for a file in two snapshots, the mirror daemon should not sync the attributes.

On the the mirror, the ctimes increase with every snapshot towards more current dates.

Given that the CephFS Mirror daemon writes the data to the mirror as a CephFS client,
it seems to make sense that data on the mirror has different / more recent ctimes (from writing).
Also, when the mirror daemon is syncing the attributes to the mirror, wouldn't this trigger an new/current ctime as well?

Right. The mirror daemon synchronizes {a,m}time, however, ctime moves forward.

So our assumption is, syncing an old ctime will actually result in a new ctime.
And thus trigger the sync of attributes over and over (at least with every snapshot synced).

The mirror daemon should be comparing ctime for a file between two snapshots in the primary cluster. If you aren't seeing that, then its probably a bug that's causing unnecessary attr sync.

So is ctime the proper parameter to test if attributes need to be synced? Or shouldn't it rather be excluded?
So is this check the right thing to do: *need_attr_sync = (cstx.stx_ctime != pstx.stx_ctime)

Is it reasonable to assume that these attribute syncs are responsible for our slow mirroring?
Or is there anything else we should look out for?

Could you share (upload) the debug mirror logs. I will have a look

And are there actually commands or logs showing us the speed of the mirroring?
We only now about sync_duration and sync_time_stamp (as in the status above).
But then, how can we actually determine the size of a snapshot or the difference between snapshots?
So one can make speed calculations for the latest sync.

Those aren't available anywhere at the moment. There were some considerations to use rbytes of snapshots (from primary), however, recursive stats (ones starting with "r", rbytes, rsize, etc..) are currently a bit buggy for snapshots. So, we decided not to rely on those. Unfortunately, we do not have a good way to provide this bit of information.

What is the general experience with mirroring performance?
In particular with bigger cephfs directories towards peta bytes.

The optimizations are a bit limited right now, especially when it comes to walking the directory tree and synchronizing file data. We have changes (under review) that adds functionality that can be used to speed up mirror daemon, but those will take a bit of time to be baked in.

Also, minor typo in the admin daemon output: "current_sycning_snap"