Ceph » Orchestrator

Answer by Valentin:

Hi Sebastian, feel free to ignore this warning. Systemd still supports KillMode=none but they decided to very slowly deprecate it. The reasons for that was that some third-party vendors somehow abused that setting which in turn caused problems during shutdown; some units just didn't want to die.
Starting with Podman v3.2 the units are using type=notify which changed many things but we found an alternative solution. Let me look it up quickly.

An alternative to KillMode=none is to remove that but add a high time out TimeoutStopSec=70. The reasoning behind is that we want to avoid between Podman and systemd trying to kill the container. We want Podman to do that, such that services have the chance to stop gracefully and that Podman can perform the necessary clean-up tasks.
type=notify is more systemd-idiomatic and what the systemd folks asked us to do in the future

Hope that helps.

Bottom line is: the warning is nothing to worry about for now. The systemd folks wanted to make it scary for the aforementioned reasons.

Quick question on this, we're seeing services with KillMode=none being killed for a restart after 10s (RestartSec=10s) during ceph orch upgrades.

It's taking iscsi containers more than 10 seconds to shutdown, so the new container being brought up is unable to access the resources it needs:

16:45:11 Upgrade: Updating iscsi.iscsi.<hostname>.nrtwwn (1/2)
16:45:11 Deploying daemon iscsi.iscsi.<hostname>.nrtwwn on <hostname>
16:45:11 Stopping Ceph iscsi.iscsi.<hostname>.nrtwwn for 4ade5f05-4ad7-4440-a0f6-00d833b6730c...
16:45:12 debug Shutdown received
16:45:23 Stopped Ceph iscsi.iscsi.<hostname>.nrtwwn for 4ade5f05-4ad7-4440-a0f6-00d833b6730c.
16:45:23 ceph-4ade5f05-4ad7-4440-a0f6-00d833b6730c@iscsi.iscsi.<hostname>.nrtwwn.service: Consumed 8h 43min 45.587s CPU time.
16:45:23,689 7f0107b81740 WARNING Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d4ade5f05\x2d4ad7\x2d4440\x2da0f6\x2d00d833b6730c.slice/ceph-4ade5f05-4ad7-4440-a0f6-00d833b6730c@iscsi.iscsi.<hostname>.nrtwwn.service
16:45:23 Starting Ceph iscsi.iscsi.<hostname>.nrtwwn for 4ade5f05-4ad7-4440-a0f6-00d833b6730c...
16:45:23 ceph-4ade5f05-4ad7-4440-a0f6-00d833b6730c@iscsi.iscsi.<hostname>.nrtwwn.service: Failed to attach to cgroup /system.slice/system-ceph\x2d4ade5f05\x2d4ad7\x2d4440\x2da0f6\x2d00d833b6730c.slice/ceph-4ade5f05-4ad7-4440-a0f6-00d833b6730c@iscsi.iscsi.<hostname>.nrtwwn.service: Device or resource busy
16:45:23 ceph-4ade5f05-4ad7-4440-a0f6-00d833b6730c@iscsi.iscsi.<hostname>.nrtwwn.service: Failed at step CGROUP spawning /bin/bash: Device or resource busy
16:45:23 ceph-4ade5f05-4ad7-4440-a0f6-00d833b6730c@iscsi.iscsi.<hostname>.nrtwwn.service: Control process exited, code=exited, status=219/CGROUP

...causing the upgrade to pause with error:

UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.iscsi.<hostname>.lofqpp on host <hostname> failed`. 

If KillMode is staying as none, should the RestartSec timeout be longer too, as looks like systemd ignores TimeoutStopSec during a service restart?

I just saw the warning about KillMode=none for unit `ceph-volume@.service` on Reef (18.2.1) on Ubuntu 22.04 LTS (running sytemd 249). Apparently the deprecation warning was introduced with 246 already, see https://github.com/systemd/systemd/blob/8a9bf03bd7e301037e32277d58adbd2eb7ce5534/NEWS#L5887.

1. FWIW, this is a non-cephadm installation using packages (https://docs.ceph.com/en/latest/install/get-packages/#debian-packages), not containers via podman.
2. Podman did change it's systemd generator / recommendations away from `KillMode=none` a long time ago: https://github.com/containers/podman/pull/8889
3. And they then switched to sdnotify to fully integrate with systemd, see https://github.com/containers/podman/pull/10557

cephadm apparently still uses a Killmode=none and outdated Type=Forking it's templates:

• https://github.com/ceph/ceph/blob/a3415f7d0f951e50c0848c1ddfb5ca503c199596/src/cephadm/cephadmlib/templates/ceph.service.j2#L25
• https://github.com/ceph/ceph/blob/a3415f7d0f951e50c0848c1ddfb5ca503c199596/src/cephadm/tests/test_unit_file.py#L96
• https://github.com/ceph/ceph/blob/a3415f7d0f951e50c0848c1ddfb5ca503c199596/src/cephadm/cephadmlib/templates/sidecar.service.j2#L22

maybe it makes sense to at least use the podman systemd-generator (https://docs.podman.io/en/latest/markdown/podman-generate-systemd.1.html) to modernize the systemd units in general?
While the generator is also somewhat deprecated in favor of the Quadlet files (https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html) it likely still helps in resolving this issue here.

we moved over to killMode=none was to avoid https://tracker.ceph.com/issues/43883 . Might be that this is no longer a problem?

Sebastian Wagner wrote:

we moved over to killMode=none was to avoid https://tracker.ceph.com/issues/43883 . Might be that this is no longer a problem?

That's totally my assumption looking at the changes done to podman itself and the recommended unit Type=Notify.
The side effect was due to systemd trying its best to track the process(es) launched by the unit (which was podman, which then spawned and maintained yet another process).
But this is all obsolete now and deserves an overhaul.

I also opened issue https://tracker.ceph.com/issues/64146 for ceph-volume still using KillMode=none.
That is totally unrelated to cephadm so I thought it might make sense to track and deal with this independently.

FWIW, the cephadm team is aware of the issue, it's just that it has been a lower priority as it's "just" a warning - an annoying one for sure, though.

A few months back I did a fairly large refactor of cephadm including adding support for a templating library. Until that was done, the "templating" of systemd unit files was rather restricted. I think that with the updated library code it would be easier to switch to using better, more recent, unit file idioms when using podman, but we also have to be careful not to break docker support.

While I don't have the bandwidth to do it myself at the moment I'd be happy to coach anyone who wants to volunteer for the task.