Bug #14839
closedsystemctl stop ceph.target doesn't stop ceph
0%
Description
This is on centos 7.2 and easy to recreate
[ubuntu@mira038 ~]$ sudo systemctl stop ceph.target
[ubuntu@mira038 ~]$ ps -eaf | grep ceph
ceph 19782 1 0 18:59 ? 00:00:03 /usr/bin/ceph-mon -f --cluster ceph --id mira038 --setuser ceph --setgroup ceph
ceph 21165 1 0 19:00 ? 00:00:07 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph
ceph 22172 1 0 19:00 ? 00:00:07 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
ceph 23199 1 0 19:01 ? 00:00:09 /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
ubuntu 25631 14444 0 19:22 pts/0 00:00:00 grep --color=auto ceph
[ubuntu@mira038 ~]$ sudo systemctl stop ceph.target
[ubuntu@mira038 ~]$ ps -eaf | grep ceph
ceph 19782 1 0 18:59 ? 00:00:03 /usr/bin/ceph-mon -f --cluster ceph --id mira038 --setuser ceph --setgroup ceph
ceph 21165 1 0 19:00 ? 00:00:07 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph
ceph 22172 1 0 19:00 ? 00:00:07 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
ceph 23199 1 0 19:01 ? 00:00:09 /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
ubuntu 25639 14444 0 19:22 pts/0 00:00:00 grep --color=auto ceph
[ubuntu@mira038 ~]$ sudo systemctl status
● mira038
State: degraded
Jobs: 0 queued
Failed: 1 units
Since: Sun 2016-02-21 17:32:18 UTC; 1 day 1h ago
CGroup: /
├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
├─system.slice
│ ├─dbus.service
│ │ └─686 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─firewalld.service
│ │ └─712 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
│ ├─lvm2-lvmetad.service
│ │ └─478 /usr/sbin/lvmetad -f
│ ├─abrtd.service
│ │ └─655 /usr/sbin/abrtd -d -s
│ ├─system-ceph\x2dmon.slice
│ │ └─ceph-mon@mira038.service
│ │ └─19782 /usr/bin/ceph-mon -f --cluster ceph --id mira038 --setuser ceph --setgroup ceph
│ ├─wpa_supplicant.service
│ │ └─895 /usr/sbin/wpa_supplicant -u -f /var/log/wpa_supplicant.log -c /etc/wpa_supplicant/wpa_supplicant.conf -u -f /var/log/wpa_supplicant.log -P /var/run/wpa_
│ ├─postfix.service
│ │ ├─1997 /usr/libexec/postfix/master -w
│ │ ├─2022 qmgr -l -t unix -u
│ │ └─9266 pickup -l -t unix -u
│ ├─crond.service
│ │ └─725 /usr/sbin/crond -n
│ ├─system-ceph\x2dosd.slice
│ │ ├─ceph-osd@1.service
│ │ │ └─22172 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
│ │ ├─ceph-osd@0.service
│ │ │ └─21165 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph
│ │ └─ceph-osd@2.service
│ │ └─23199 /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
│ ├─system-serial\x2dgetty.slice
│ │ └─serial-getty@ttyS2.service
│ │ └─1435 /sbin/agetty --keep-baud 115200 38400 9600 ttyS2 vt220
│ ├─atd.service
│ │ └─727 /usr/sbin/atd -f
│ ├─systemd-journald.service
from logs:
Feb 22 19:22:32 mira038 systemd: Stopped target ceph target allowing to start/stop all ceph*.service instances at once..service instances at once.
Feb 22 19:22:32 mira038 systemd: Stopping ceph target allowing to start/stop all ceph*
Feb 22 19:22:42 mira038 systemd: Stopped target ceph target allowing to start/stop all ceph*@.service instances at once.
Updated by Vasu Kulkarni about 8 years ago
This is on master, but same seen in testing on infernalis/jewel and ceph-deploy tests logs on centos show this issue but somehow the test passed maybe due to return code from shell.
[ubuntu@mira038 ~]$ sudo ceph -v ceph version 10.0.3-2191-g8976412 (89764124b3e1ebfbf7d794fa3d5d5681ca64eb43)
Updated by Vasu Kulkarni about 8 years ago
Boris told me try below which worked
<branto> workaround is to run 'systemctl stop ceph-mon.target ceph-osd.target ceph-mds.target ceph-radosgw.target'
[ubuntu@mira038 ~]$ sudo systemctl stop ceph-osd.target [ubuntu@mira038 ~]$ ps -eaf | grep ceph ceph 19782 1 0 18:59 ? 00:00:04 /usr/bin/ceph-mon -f --cluster ceph --id mira038 --setuser ceph --setgroup ceph ubuntu 25770 14444 0 19:58 pts/0 00:00:00 grep --color=auto ceph [ubuntu@mira038 ~]$ sudo systemctl stop ceph-mon.target [ubuntu@mira038 ~]$ ps -eaf | grep ceph ubuntu 25783 14444 0 19:58 pts/0 00:00:00 grep --color=auto ceph
Updated by Boris Ranto about 8 years ago
Few more notes:
This was probably introduced by [1]. We should probably consider using some new dependency layout to fix this. The list of options is available at [2].
[1] https://github.com/ceph/ceph/pull/6276
[2] https://www.freedesktop.org/software/systemd/man/systemd.unit.html
Updated by Boris Ranto about 8 years ago
@Vasu Kulkarni: Not sure, I'm really not a systemd expert at the moment. I can try to ask some people that work on systemd for some help, though.
Updated by Yuri Weinstein about 8 years ago
- Release set to jewel
- ceph-qa-suite ceph-deploy added
Updated by Vasu Kulkarni almost 8 years ago
- Priority changed from High to Urgent
This is not fixed in ceph master as well, the behaviour is diffent based on the node which has mon+osd role vs node which has only osd role. If you need a system in this state let me know.
Updated by Yuri Weinstein over 7 years ago
- Assignee set to Vasu Kulkarni
Updated by Sage Weil almost 7 years ago
- Status changed from New to 12
I think https://github.com/ceph/ceph/pull/15835 addresses this problem?
Updated by Boris Ranto almost 7 years ago
This should actually be fixed for quite some time (we had a couple of issues with the way we handled systemd in the rpms). The issue in the PR was introduced quite recently when we added a dependency from ceph-osd to ceph-mon.