Backport #21780
closedxenial 16.04/ after jewel -> luminous upgrade Failed to stop ceph.target: Transaction order is cyclic
Description
Steps:
1) install stable jewel release
2) upgrade to dev branch of luminous
3) try starting/stopping various ceph daemons
on one of the nodes following issue is seen
2017-10-12T23:33:54.066 INFO:teuthology.orchestra.run.vpm199:Running: 'sudo systemctl start ceph-mgr@vpm199.service' 2017-10-12T23:33:58.153 INFO:teuthology.orchestra.run.vpm141:Running: 'sudo ps -eaf | grep ceph' 2017-10-12T23:33:58.180 INFO:teuthology.orchestra.run.vpm141.stdout:ubuntu 25706 16821 0 23:33 ? 00:00:00 bash -c sudo ps -eaf | grep ceph 2017-10-12T23:33:58.180 INFO:teuthology.orchestra.run.vpm141.stdout:ubuntu 25708 25706 0 23:33 ? 00:00:00 grep ceph 2017-10-12T23:33:58.180 INFO:teuthology.orchestra.run.vpm141.stdout:ceph 26662 1 0 23:15 ? 00:00:03 /usr/bin/ceph-osd -f --cluster ceph --id 3 --setuser ceph --setgroup ceph 2017-10-12T23:33:58.180 INFO:teuthology.orchestra.run.vpm141.stdout:ceph 28126 1 0 23:15 ? 00:00:03 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph 2017-10-12T23:33:58.180 INFO:teuthology.orchestra.run.vpm141.stdout:ceph 29584 1 0 23:15 ? 00:00:02 /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph 2017-10-12T23:33:58.181 INFO:teuthology.orchestra.run.vpm141:Running: 'sudo systemctl list-units | grep ceph' 2017-10-12T23:33:58.289 INFO:teuthology.orchestra.run.vpm141.stdout: var-lib-ceph-osd-ceph\x2d3.mount loaded active mounted /var/lib/ceph/osd/ceph-3 2017-10-12T23:33:58.289 INFO:teuthology.orchestra.run.vpm141.stdout: var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout: var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout: ceph-osd@3.service loaded active running Ceph object storage daemon 2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout: ceph-osd@4.service loaded active running Ceph object storage daemon 2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout: ceph-osd@5.service loaded active running Ceph object storage daemon 2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout: system-ceph\x2ddisk.slice loaded active active system-ceph\x2ddisk.slice 2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout: system-ceph\x2dosd.slice loaded active active system-ceph\x2dosd.slice 2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout: ceph-mds.target loaded active active ceph target allowing to start/stop all ceph-mds@.service instances at once 2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout: ceph-mon.target loaded active active ceph target allowing to start/stop all ceph-mon@.service instances at once 2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout: ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once 2017-10-12T23:33:58.291 INFO:teuthology.orchestra.run.vpm141.stdout: ceph-radosgw.target loaded active active ceph target allowing to start/stop all ceph-radosgw@.service instances at once 2017-10-12T23:33:58.291 INFO:teuthology.orchestra.run.vpm141.stdout: ceph.target loaded active active ceph target allowing to start/stop all ceph*@.service instances at once 2017-10-12T23:33:58.291 INFO:tasks.systemd: var-lib-ceph-osd-ceph\x2d3.mount loaded active mounted /var/lib/ceph/osd/ceph-3 var-lib-ceph-osd-ceph\x2d4.mount loaded active mounted /var/lib/ceph/osd/ceph-4 var-lib-ceph-osd-ceph\x2d5.mount loaded active mounted /var/lib/ceph/osd/ceph-5 ceph-osd@3.service loaded active running Ceph object storage daemon ceph-osd@4.service loaded active running Ceph object storage daemon ceph-osd@5.service loaded active running Ceph object storage daemon system-ceph\x2ddisk.slice loaded active active system-ceph\x2ddisk.slice system-ceph\x2dosd.slice loaded active active system-ceph\x2dosd.slice ceph-mds.target loaded active active ceph target allowing to start/stop all ceph-mds@.service instances at once ceph-mon.target loaded active active ceph target allowing to start/stop all ceph-mon@.service instances at once ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once ceph-radosgw.target loaded active active ceph target allowing to start/stop all ceph-radosgw@.service instances at once ceph.target loaded active active ceph target allowing to start/stop all ceph*@.service instances at once 2017-10-12T23:33:58.291 INFO:tasks.systemd:Ceph services in failed state 2017-10-12T23:33:58.291 INFO:tasks.systemd:Stopping all Ceph services 2017-10-12T23:33:58.291 INFO:teuthology.orchestra.run.vpm141:Running: 'sudo systemctl stop ceph.target' 2017-10-12T23:33:58.310 INFO:teuthology.orchestra.run.vpm141.stderr:Failed to stop ceph.target: Transaction order is cyclic. See system logs for details. 2017-10-12T23:33:58.310 INFO:teuthology.orchestra.run.vpm141.stderr:See system logs and 'systemctl status ceph.target' for details. 2017-10-12T23:33:58.311 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last):
Updated by Nathan Cutler over 6 years ago
Note: this was fixed in master by https://github.com/ceph/ceph/pull/15835 (included in luminous v12.2.0 release)
From the bug description it appears you are not rebooting the nodes:
1) install stable jewel release
2) upgrade to dev branch of luminous
3) try starting/stopping various ceph daemons
Please add a step:
2.5) reboot the cluster nodes
Does that make the issue go away?
A proper fix will be to backport 357dfa5954a3386c92dee3289a669a2c08d41eda to jewel as well.
Updated by Nathan Cutler over 6 years ago
- Tracker changed from Bug to Backport
- Status changed from New to In Progress
- Assignee set to Nathan Cutler
- Release set to jewel
jewel backport PR: https://github.com/ceph/ceph/pull/18290
Updated by Vasu Kulkarni over 6 years ago
we restart the services so reboot is not required, I believe we also missed to update one of the osd node upgrade so we found the jewel bug which is still good :)
Updated by Nathan Cutler over 6 years ago
Vasu Kulkarni wrote:
we restart the services so reboot is not required
I meant that as a question, i.e. can you try the reboot to see if the error persists after the reboot.
Updated by Nathan Cutler over 6 years ago
- Status changed from In Progress to Duplicate
Duplicate of #21478
Updated by Nathan Cutler over 6 years ago
- Is duplicate of Backport #21478: jewel: systemd: Add explicit Before=ceph.target added