Project

General

Profile

Bug #21477

systemd: Add explicit Before=ceph.target

Added by Nathan Cutler over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

The PartOf= and WantedBy= directives in the various systemd
unit files and targets create the following logical hierarchy:

- ceph.target
- ceph-fuse.target
- ceph-fuse@.service
- ceph-mds.target
- ceph-mds@.service
- ceph-mgr.target
- ceph-mgr@.service
- ceph-mon.target
- ceph-mon@.service
- ceph-osd.target
- ceph-osd@.service
- ceph-radosgw.target
- ceph-radosgw@.service
- ceph-rbd-mirror.target
- ceph-rbd-mirror@.service

Additionally, the ceph-{fuse,mds,mon,osd,radosgw,rbd-mirror}
targets have WantedBy=multi-user.target. This gives the
following behaviour:

- `systemctl {start,stop,restart}` of any target will restart
all dependent services (e.g.: `systemctl restart ceph.target`
will restart all services; `systemctl restart ceph-mon.target`
will restart all the mons, and so forth).
- `systemctl {enable,disable}` for the second level targets
(ceph-mon.target etc.) will cause depenent services to come
up on boot, or not (of course the individual services can
be enabled or disabled as well - for a service to start
on boot, both the service and its target must be enabled;
disabling either will cause the service to be disabled).
- `systemctl {enable,disable} ceph.target` has no effect on
whether or not services come up at boot; if the second level
targets and services are enabled, they'll start regardless of
whether ceph.target is enabled. This is due to the second
level targets all having WantedBy=multi-user.target.
- The OSDs will always start regardless of ceph-osd.target
(unless they are explicitly masked), thanks to udev magic.

So far, so good. Except, several users have encountered
services not starting with the following error:

Failed to start : Transaction order is
cyclic. See system logs for details.

I've not been able to reproduce this myself in such a way as to
cause OSDs to fail to start, but I have managed to get systemd
into that same confused state, as follows:

- Disable ceph.target, ceph-mon.target, ceph-osd.target,
ceph-mon@$(hostname).service and all ceph-osd instances.
- Re-enable all of the above.

At this point, everything is fine, but if I then subseqently
disable ceph.target, then try `systemctl restart ceph.target`,
I get "Failed to restart ceph.target: Transaction order is cyclic.
See system logs for details."

Explicitly adding Before=ceph.target to each second level target
prevents systemd from becoming confused in this situation.


Related issues

Copied to Ceph - Backport #21478: jewel: systemd: Add explicit Before=ceph.target Resolved

History

#1 Updated by Nathan Cutler over 6 years ago

master PR: https://github.com/ceph/ceph/pull/15835

We have seen the "cyclic dependency" error appear after applying http://tracker.ceph.com/issues/19910 and http://tracker.ceph.com/issues/21035 (both of which are queued for jewel v10.2.10) to a jewel 10.2.9 cluster.

#2 Updated by Nathan Cutler over 6 years ago

  • Copied to Backport #21478: jewel: systemd: Add explicit Before=ceph.target added

#3 Updated by Kefu Chai over 6 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF