Project

General

Profile

Actions
Copy link

Backport #21780

closed

xenial 16.04/ after jewel -> luminous upgrade Failed to stop ceph.target: Transaction order is cyclic

Added by Vasu Kulkarni over 6 years ago. Updated over 6 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
Nathan Cutler
Target version:
-
Release:
jewel
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Steps:

1) install stable jewel release
2) upgrade to dev branch of luminous
3) try starting/stopping various ceph daemons
on one of the nodes following issue is seen

2017-10-12T23:33:54.066 INFO:teuthology.orchestra.run.vpm199:Running: 'sudo systemctl start ceph-mgr@vpm199.service'
2017-10-12T23:33:58.153 INFO:teuthology.orchestra.run.vpm141:Running: 'sudo ps -eaf | grep ceph'
2017-10-12T23:33:58.180 INFO:teuthology.orchestra.run.vpm141.stdout:ubuntu   25706 16821  0 23:33 ?        00:00:00 bash -c sudo ps -eaf | grep ceph
2017-10-12T23:33:58.180 INFO:teuthology.orchestra.run.vpm141.stdout:ubuntu   25708 25706  0 23:33 ?        00:00:00 grep ceph
2017-10-12T23:33:58.180 INFO:teuthology.orchestra.run.vpm141.stdout:ceph     26662     1  0 23:15 ?        00:00:03 /usr/bin/ceph-osd -f --cluster ceph --id 3 --setuser ceph --setgroup ceph
2017-10-12T23:33:58.180 INFO:teuthology.orchestra.run.vpm141.stdout:ceph     28126     1  0 23:15 ?        00:00:03 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph
2017-10-12T23:33:58.180 INFO:teuthology.orchestra.run.vpm141.stdout:ceph     29584     1  0 23:15 ?        00:00:02 /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph
2017-10-12T23:33:58.181 INFO:teuthology.orchestra.run.vpm141:Running: 'sudo systemctl list-units | grep ceph'
2017-10-12T23:33:58.289 INFO:teuthology.orchestra.run.vpm141.stdout:  var-lib-ceph-osd-ceph\x2d3.mount                                                    loaded    active mounted   /var/lib/ceph/osd/ceph-3
2017-10-12T23:33:58.289 INFO:teuthology.orchestra.run.vpm141.stdout:  var-lib-ceph-osd-ceph\x2d4.mount                                                    loaded    active mounted   /var/lib/ceph/osd/ceph-4
2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout:  var-lib-ceph-osd-ceph\x2d5.mount                                                    loaded    active mounted   /var/lib/ceph/osd/ceph-5
2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout:  ceph-osd@3.service                                                                  loaded    active running   Ceph object storage daemon
2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout:  ceph-osd@4.service                                                                  loaded    active running   Ceph object storage daemon
2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout:  ceph-osd@5.service                                                                  loaded    active running   Ceph object storage daemon
2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout:  system-ceph\x2ddisk.slice                                                           loaded    active active    system-ceph\x2ddisk.slice
2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout:  system-ceph\x2dosd.slice                                                            loaded    active active    system-ceph\x2dosd.slice
2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout:  ceph-mds.target                                                                     loaded    active active    ceph target allowing to start/stop all ceph-mds@.service instances at once
2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout:  ceph-mon.target                                                                     loaded    active active    ceph target allowing to start/stop all ceph-mon@.service instances at once
2017-10-12T23:33:58.290 INFO:teuthology.orchestra.run.vpm141.stdout:  ceph-osd.target                                                                     loaded    active active    ceph target allowing to start/stop all ceph-osd@.service instances at once
2017-10-12T23:33:58.291 INFO:teuthology.orchestra.run.vpm141.stdout:  ceph-radosgw.target                                                                 loaded    active active    ceph target allowing to start/stop all ceph-radosgw@.service instances at once
2017-10-12T23:33:58.291 INFO:teuthology.orchestra.run.vpm141.stdout:  ceph.target                                                                         loaded    active active    ceph target allowing to start/stop all ceph*@.service instances at once
2017-10-12T23:33:58.291 INFO:tasks.systemd:  var-lib-ceph-osd-ceph\x2d3.mount                                                    loaded    active mounted   /var/lib/ceph/osd/ceph-3
  var-lib-ceph-osd-ceph\x2d4.mount                                                    loaded    active mounted   /var/lib/ceph/osd/ceph-4
  var-lib-ceph-osd-ceph\x2d5.mount                                                    loaded    active mounted   /var/lib/ceph/osd/ceph-5
  ceph-osd@3.service                                                                  loaded    active running   Ceph object storage daemon
  ceph-osd@4.service                                                                  loaded    active running   Ceph object storage daemon
  ceph-osd@5.service                                                                  loaded    active running   Ceph object storage daemon
  system-ceph\x2ddisk.slice                                                           loaded    active active    system-ceph\x2ddisk.slice
  system-ceph\x2dosd.slice                                                            loaded    active active    system-ceph\x2dosd.slice
  ceph-mds.target                                                                     loaded    active active    ceph target allowing to start/stop all ceph-mds@.service instances at once
  ceph-mon.target                                                                     loaded    active active    ceph target allowing to start/stop all ceph-mon@.service instances at once
  ceph-osd.target                                                                     loaded    active active    ceph target allowing to start/stop all ceph-osd@.service instances at once
  ceph-radosgw.target                                                                 loaded    active active    ceph target allowing to start/stop all ceph-radosgw@.service instances at once
  ceph.target                                                                         loaded    active active    ceph target allowing to start/stop all ceph*@.service instances at once

2017-10-12T23:33:58.291 INFO:tasks.systemd:Ceph services in failed state
2017-10-12T23:33:58.291 INFO:tasks.systemd:Stopping all Ceph services
2017-10-12T23:33:58.291 INFO:teuthology.orchestra.run.vpm141:Running: 'sudo systemctl stop ceph.target'
2017-10-12T23:33:58.310 INFO:teuthology.orchestra.run.vpm141.stderr:Failed to stop ceph.target: Transaction order is cyclic. See system logs for details.
2017-10-12T23:33:58.310 INFO:teuthology.orchestra.run.vpm141.stderr:See system logs and 'systemctl status ceph.target' for details.
2017-10-12T23:33:58.311 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):

Related issues 1 (0 open — 1 closed)

Is duplicate of Ceph - Backport #21478: jewel: systemd: Add explicit Before=ceph.targetResolvedNathan CutlerActions
Actions
Copy link
 #3

Updated by Yuri Weinstein over 6 years ago

  • Project changed from devops to Ceph
Actions
Copy link
 #4

Updated by Nathan Cutler over 6 years ago

Note: this was fixed in master by https://github.com/ceph/ceph/pull/15835 (included in luminous v12.2.0 release)

From the bug description it appears you are not rebooting the nodes:

1) install stable jewel release
2) upgrade to dev branch of luminous
3) try starting/stopping various ceph daemons

Please add a step:

2.5) reboot the cluster nodes

Does that make the issue go away?

A proper fix will be to backport 357dfa5954a3386c92dee3289a669a2c08d41eda to jewel as well.

Actions
Copy link
 #5

Updated by Nathan Cutler over 6 years ago

  • Tracker changed from Bug to Backport
  • Status changed from New to In Progress
  • Assignee set to Nathan Cutler
  • Release set to jewel
Actions
Copy link
 #6

Updated by Vasu Kulkarni over 6 years ago

we restart the services so reboot is not required, I believe we also missed to update one of the osd node upgrade so we found the jewel bug which is still good :)

Actions
Copy link
 #7

Updated by Nathan Cutler over 6 years ago

Vasu Kulkarni wrote:

we restart the services so reboot is not required

I meant that as a question, i.e. can you try the reboot to see if the error persists after the reboot.

Actions
Copy link
 #8

Updated by Nathan Cutler over 6 years ago

  • Status changed from In Progress to Duplicate

Duplicate of #21478

Actions
Copy link
 #9

Updated by Nathan Cutler over 6 years ago

  • Is duplicate of Backport #21478: jewel: systemd: Add explicit Before=ceph.target added
Actions
Copy link

Also available in: Atom PDF