Ceph » CephFS

The time consumption is part of the fetching the list of cancelled clones and issuing subvolume rm ? If that's the case, we can provide an api and do it internally. I am not sure how much time it would save though.

Kotresh Hiremath Ravishankar wrote:

The time consumption is part of the fetching the list of cancelled clones and issuing subvolume rm ?

Just for subvolume rm on ~4080 subvolumes - that's ~1.7s per subvolume removal.

If that's the case, we can provide an api and do it internally. I am not sure how much time it would save though.

That could possibly be driven by a separate thread. We'll have to check with csi folks w.r.t. using that and also know about the completion status.

I further looked into the snippet of the script used. It is as below.

For cancellation:
--------------------
sudo ceph fs subvolume ls  --vol_name cephfs --group_name csi  2>&1
sudo ceph fs clone status --clone_name csi-vol-7b664dd3-4cf4-11ee-94d9-0a580a8e0626 --vol_name cephfs --group_name csi  > /tmp/tmp.GlhVs6lsfh 2>&1
sudo ceph fs clone status --clone_name csi-vol-fafa7c7f-5474-4d85-b777-8a7bac8eb908 --vol_name cephfs --group_name csi  > /tmp/tmp.GlhVs6lsfh 2>&1
sudo ceph fs clone cancel --clone_name csi-vol-fafa7c7f-5474-4d85-b777-8a7bac8eb908 --vol_name cephfs --group_name csi  > /tmp/tmp.GlhVs6lsfh 2>&1
sudo ceph fs clone status --clone_name csi-vol-3897be6d-eaf0-4220-83a1-c36b39a085b2 --vol_name cephfs --group_name csi  > /tmp/tmp.GlhVs6lsfh 2>&1
sudo ceph fs clone cancel --clone_name csi-vol-3897be6d-eaf0-4220-83a1-c36b39a085b2 --vol_name cephfs --group_name csi  > /tmp/tmp.GlhVs6lsfh 2>&1
sudo ceph fs clone status --clone_name csi-vol-ff75b068-4ce3-11ee-9c69-0a580a8e080a --vol_name cephfs --group_name csi  > /tmp/tmp.GlhVs6lsfh 2>&1
sudo ceph fs clone status --clone_name csi-vol-90fc0660-479c-4106-9301-5c3de635fcc0 --vol_name cephfs --group_name csi  > /tmp/tmp.GlhVs6lsfh 2>&1
sudo ceph fs clone cancel --clone_name csi-vol-90fc0660-479c-4106-9301-5c3de635fcc0 --vol_name cephfs --group_name csi  > /tmp/tmp.GlhVs6lsfh 2>&1
sudo ceph fs clone status --clone_name csi-vol-c6c56d38-69bd-4905-8e73-6b992cced02d --vol_name cephfs --group_name csi  > /tmp/tmp.GlhVs6lsfh 2>&1
...
...

For clone/subvolume deletion:
-----------------------------
sudo ceph fs subvolume ls  --vol_name cephfs --group_name csi  2>&1
sudo ceph fs clone status --clone_name csi-vol-7b664dd3-4cf4-11ee-94d9-0a580a8e0626 --vol_name cephfs --group_name csi  > /tmp/tmp.HjuB6hDYKO 2>&1
sudo ceph fs subvolume rm --sub_name csi-vol-7b664dd3-4cf4-11ee-94d9-0a580a8e0626 --vol_name cephfs --group_name csi --force > /tmp/tmp.HjuB6hDYKO 2>&1
sudo ceph fs clone status --clone_name csi-vol-fafa7c7f-5474-4d85-b777-8a7bac8eb908 --vol_name cephfs --group_name csi  > /tmp/tmp.HjuB6hDYKO 2>&1
sudo ceph fs subvolume rm --sub_name csi-vol-fafa7c7f-5474-4d85-b777-8a7bac8eb908 --vol_name cephfs --group_name csi --force > /tmp/tmp.HjuB6hDYKO 2>&1
sudo ceph fs clone status --clone_name csi-vol-3897be6d-eaf0-4220-83a1-c36b39a085b2 --vol_name cephfs --group_name csi  > /tmp/tmp.HjuB6hDYKO 2>&1
sudo ceph fs subvolume rm --sub_name csi-vol-3897be6d-eaf0-4220-83a1-c36b39a085b2 --vol_name cephfs --group_name csi --force > /tmp/tmp.HjuB6hDYKO 2>&1
sudo ceph fs clone status --clone_name csi-vol-ff75b068-4ce3-11ee-9c69-0a580a8e080a --vol_name cephfs --group_name csi  > /tmp/tmp.HjuB6hDYKO 2>&1
sudo ceph fs subvolume rm --sub_name csi-vol-ff75b068-4ce3-11ee-9c69-0a580a8e080a --vol_name cephfs --group_name csi --force > /tmp/tmp.HjuB6hDYKO 2>&1
sudo ceph fs clone status --clone_name csi-vol-90fc0660-479c-4106-9301-5c3de635fcc0 --vol_name cephfs --group_name csi  > /tmp/tmp.HjuB6hDYKO 2>&1
sudo ceph fs subvolume rm --sub_name csi-vol-90fc0660-479c-4106-9301-5c3de635fcc0 --vol_name cephfs --group_name csi --force > /tmp/tmp.HjuB6hDYKO 2>&1
sudo ceph fs clone status --clone_name csi-vol-c6c56d38-69bd-4905-8e73-6b992cced02d --vol_name cephfs --group_name csi  > /tmp/tmp.HjuB6hDYKO 2>&1
...
...

So one 'clone status' and 'subvolume rm' command for each deletion of cancelled clone. So, having a single api could be helpful potentially saving network time/lock time? when thousands of cancelled clones needs to be deleted. I think this is worth exploring. But we maintain the list of pending clones but not the cancelled clones. So might need to add support for that in order to achieve this.

Also, more time is consumed (around 4-5 hrs) for cancelling 4k pending clones. Not sure, we should also look into this. But this is tricky as we don't know the list of the clones that needs to be cancelled among all pending clones via one api.

Thanks,
Kotresh H R

Ah, I forgot, the most important part for me to write this script was to give the CU a progress counter so we all actually know where we are and how long it still will going to take, without this, we are blind and I have CU cancel those long standing operations because it seems to not move further.

Kotresh Hiremath Ravishankar wrote:

I further looked into the snippet of the script used. It is as below.

[...]

So one 'clone status' and 'subvolume rm' command for each deletion of cancelled clone. So, having a single api could be helpful potentially saving network time/lock time? when thousands of cancelled clones needs to be deleted. I think this is worth exploring. But we maintain the list of pending clones but not the cancelled clones. So might need to add support for that in order to achieve this.

Right now canceling a clone removes the index pointer. One option could be to leave the index pointer as it is and teach the cloner thread to skip canceled clones. Furthermore, a separate thread would sweep the indexes and clean out canceled clones. Or, have a separate tracking directory where the canceled indexes are moved. Doing this in bulk would save lot of context switch calls.

Also, more time is consumed (around 4-5 hrs) for cancelling 4k pending clones. Not sure, we should also look into this. But this is tricky as we don't know the list of the clones that needs to be cancelled among all pending clones via one api.

Yeh. We can either support cancelling all pending clones (checkpoint the last pending clone and initiate cancel till that clone - to avoid endlessly scanning if more clones are sheduled) or a single clone (which is what's supported right now).

Raimund Sacherer wrote:

Ah, I forgot, the most important part for me to write this script was to give the CU a progress counter so we all actually know where we are and how long it still will going to take, without this, we are blind and I have CU cancel those long standing operations because it seems to not move further.

A bunch of the async jobs in volumes plugin is getting hooked withe a progress counters that nicely shows up in `ceph status`.

Thanks for looking into this!