/usr/lib/sysctl.d/90-ceph-osd.conf getting installed in container, rendering it ineffective
The file /usr/lib/sysctl.d/90-ceph-osd.conf has the following contents:
fs.aio-max-nr = 1048576 kernel.pid_max = 4194304
This is very important to have on all OSD nodes. However, since this file is owned by the ceph-osd package, and that package is installed in the container image, and not in the host system, it no longer takes effect. This causes problems on OSDs nodes with larger numbers of disks.
One option would be for cephadm to manage this file, ensuring that it gets created on OSD nodes when OSDs are created, and removing it when a node no longer has any OSDs.
Another option would be to move the file to ceph-base. Then it would be installed (in the host system) on all cluster nodes, regardless of whether the node has OSDs or not.
Yet another option would be to move the file into ceph-common. In this scenario, the file would be installed on both cluster nodes and client nodes as well.