Project

General

Profile

Actions
Copy link

Bug #58242

closed

cephadm doesn't communicate with containers that failed initial start but were successfully restarted later by systemd

Added by Voja Molani over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Redouane Kachach Elhichou
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
backport_processed
Backport:
quincy, pacific
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
Ceph - v17.2.5
ceph-qa-suite:
Pull request ID:
49728
Crash signature (v1):
Crash signature (v2):

Description

If, after a host restart, the initial start of a container failed for whatever reason (perhaps because the systemd start timeout TimeoutStartSec was exceeded) then cephadm does not detect the container as running even if a subsequent automatic restart by systemd successfully started the container.

To reproduce:
  • Restart the host that runs a lot of services and look at the log for how long it takes to start the various service containers.
  • Edit the ceph container systemd service file at /etc/systemd/system/ceph-<fsid><at>.service and change TimeoutStartSec to be just a few seconds less than the previously recorded longest time to start ceph services when all containers are starting at the same time. The idea is to have it long enough to allow some/most containers to start but cause a start timeout on others.
  • Restart the host.
  • Since the TimeoutStartSec now makes some containers time out on start when all the containers are starting at the same time, systemd will terminate + restart later the containers that timed out starting.
  • After systemd tries to later restart the containers that timed out, hopefully now the system is less busy and the container will start faster, therefore not exceeding the start timeout.
  • The containers that systemd restarted later, and that now did not exceed the start timeout because the system was less busy, are running.
  • Do a ceph orch ps and witness the error in the "STATUS" field for those containers that did not start on the initial start attempt (but are now running just fine after systemd restarted them).

If there is some other way besides TimeoutStartSec to make the initial container start after boot to fail but still allow the subsequent automatic restart by systemd to succeed then use that for reproducing.
Only thing that seems to matter is that the container needs to fail the initial start and successfully start on the later automatic restart by systemd. cephadm does not detect such containers as running in ceph orch ls or ceph orch ps or other commands, even if ceph -s shows that all mons, mgrs, osds etc are running just fine. Such containers seem to be invisible to cephadm.

cephadm needs to handle the case where a container failed the initial start attempt but successfully started on the later automatic restart attempt by systemd.

Files

systemctl-ceph.log (96.1 KB) systemctl-ceph.log Voja Molani, 01/18/2023 04:02 AM

Related issues 4 (1 open — 3 closed)

Related to Orchestrator - Bug #51361: KillMode=none is deprecatedNew

Actions
Related to Orchestrator - Bug #58241: Systemd unit file TimeoutStartSec of 120 seconds is too lowResolvedRedouane Kachach Elhichou

Actions
Copied to Orchestrator - Backport #58777: pacific: cephadm doesn't communicate with containers that failed initial start but were successfully restarted later by systemdResolvedAdam KingActions
Copied to Orchestrator - Backport #58778: quincy: cephadm doesn't communicate with containers that failed initial start but were successfully restarted later by systemdResolvedAdam KingActions
Actions
Copy link

Also available in: Atom PDF