Ceph » ceph-volume

On debian 11, kernel 5.10.17-v8+, aarch64, on rpi4, podman 3.0.1 mounts volumes as source device '/dev/root':

root@node1:~# podman run --rm -v /var/lib/ceph:/test -ti alpine /bin/sh
/ # grep test /proc/mounts
/dev/root /test ext4 rw,noatime 0 0

That device is a pseudo-device, that should be interpolated from the /proc/cmdile:

/ # cat /proc/cmdline
coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 smsc95xx.macaddr=E4:5F:01:0C:03:A8 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 console=ttyS0,115200 console=tty1 root=PARTUUID=ff28b103-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

There the 'root=' option gives an identifier (it's actually more complex, as it might be that the root filesystem on the cmdline changed after... I did not find a very percise way of getting the actual device it should be, but we might not need it).

The issue is that when running the ceph container (deploying ceph with cephadm), ceph-volume will see that '/dev/root' does not point to a real device directly (as in it does not exist in that path), and will then mount tmpfs on top of the osd folder when initializing it, an example of the 'unit.run' shell script command:

#/usr/bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --name ceph-838df4f6-a854-11eb-9f8b-e45f010c03a8-mon.node1 -d --conmon-pidfile /run/ceph-838df4f6-a854-11eb-9f8b-e45f010c03a8@mon.node1.service-pid --cidfile /run/ceph-838df4f6-a854-11eb-9f8b-e45f010c03a8@mon.node1.service-cid -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.1 -e NODE_NAME=node1 -v /var/run/ceph/838df4f6-a854-11eb-9f8b-e45f010c03a8:/var/run/ceph:z -v /var/log/ceph/838df4f6-a854-11eb-9f8b-e45f010c03a8:/var/log/ceph:z -v /var/lib/ceph/838df4f6-a854-11eb-9f8b-e45f010c03a8/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/838df4f6-a854-11eb-9f8b-e45f010c03a8/mon.node1:/var/lib/ceph/mon/ceph-node1:z -v /var/lib/ceph/838df4f6-a854-11eb-9f8b-e45f010c03a8/mon.node1/config:/etc/ceph/ceph.conf:z -v /dev:/dev -v /run/udev:/run/udev docker.io/ceph/ceph:v16.2.1 -n mon.node1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix="debug " --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true

This could be solved in several ways, two options I see are:

• Introducing a --no-tmpfs flag and use that from cephadm (as in any case we will most probably not want to use tmpfs when running the container that run ceph-mon, as it will revert any change it does to the fs when the container exits and before the osd starts). This flag was added already proposed and reverted before.
• Adding a special treatment of the '/dev/root' source when parsing /proc/mounts, this could be done in two ways:
  • Just considering it a "real" path, and skip tmpfs if it is found.
  • Trying to properly figure out what filesystem is the root one (could be really tricky and error prone).
• Running ceph-volume lvm activate and the ceph osd in the same container, so the tmpfs mount does not get cleaned up on container exit (something like podman run ... bash -c "ceph-volume lvm activate ... && ceph ...")
• Do not remove the ceph-lvm container and reuse it somehow when running the osd (podman pods?)

The easiest I think would be to change ceph-volume to treat /dev/root mounts as "real" mounts and skip tmpfs, I'll send a patch for this soon.
But the more useful might be to run ceph-volume lvm activate and the osd in the same container, I might send that one too.

I'm relatively new to ceph, so I'm probably missing a lot of context, I'll be happy to help in any way though.

Thanks!

Some info:

root@node1:~# uname -a
Linux node1 5.10.17-v8+ #1403 SMP PREEMPT Mon Feb 22 11:37:54 GMT 2021 aarch64 GNU/Linux
root@node1:~# cat /etc/debian_version
bullseye/sid
root@node1:~# podman --version
podman version 3.0.1
root@node1:~# conmon --version
conmon version 2.0.25
commit: unknown

Using ceph container images docker.io/ceph/ceph:v16.2.1 (aarch64)