Project

General

Profile

Bug #17826

ceph-init syntax error breaks package remove, daemon stop

Added by David Galloway 12 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
11/08/2016
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

From: http://qa-proxy.ceph.com/teuthology/sage-2016-11-08_02:52:28-rados-wip-sage-testing---basic-smithi/531522/worker.log

2016-11-08T19:10:41.412 INFO:teuthology.kill:2016-11-08 19:10:41,412.412 INFO:teuthology.orchestra.run.smithi057.stdout:You might want to run 'apt-get -f install' to correct these.
2016-11-08T19:10:41.412 INFO:teuthology.kill:2016-11-08 19:10:41,412.412 INFO:teuthology.orchestra.run.smithi057.stdout:The following packages have unmet dependencies:
2016-11-08T19:10:41.413 INFO:teuthology.kill:2016-11-08 19:10:41,412.412 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr : Depends: ceph-base (= 11.0.2-1289-g036e761-1trusty) but it is not installed
2016-11-08T19:10:41.416 INFO:teuthology.kill:2016-11-08 19:10:41,415.415 INFO:teuthology.orchestra.run.smithi057.stderr:E: Unmet dependencies. Try using -f.
2016-11-08T19:10:41.417 INFO:teuthology.kill:2016-11-08 19:10:41,417.417 ERROR:teuthology.parallel:Exception in parallel execution
2016-11-08T19:10:41.418 INFO:teuthology.kill:Traceback (most recent call last):
2016-11-08T19:10:41.418 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 83, in __exit__
2016-11-08T19:10:41.418 INFO:teuthology.kill:    for result in self:
2016-11-08T19:10:41.418 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
2016-11-08T19:10:41.418 INFO:teuthology.kill:    resurrect_traceback(result)
2016-11-08T19:10:41.418 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
2016-11-08T19:10:41.418 INFO:teuthology.kill:    return func(*args, **kwargs)
2016-11-08T19:10:41.418 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 454, in _remove_deb
2016-11-08T19:10:41.418 INFO:teuthology.kill:    'autoremove',
2016-11-08T19:10:41.419 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 192, in run
2016-11-08T19:10:41.419 INFO:teuthology.kill:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2016-11-08T19:10:41.419 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 403, in run
2016-11-08T19:10:41.419 INFO:teuthology.kill:    r.wait()
2016-11-08T19:10:41.419 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 166, in wait
2016-11-08T19:10:41.419 INFO:teuthology.kill:    label=self.label)
2016-11-08T19:10:41.419 INFO:teuthology.kill:CommandFailedError: Command failed on smithi057 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove'
2016-11-08T19:10:41.419 INFO:teuthology.kill:2016-11-08 19:10:41,417.417 ERROR:teuthology.nuke:Could not nuke {'ubuntu@smithi057.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD4MKWct6JS0SIfEJOJopEb+e9kvl57af0a8LKuckHUsF0wA5AH9q5hUZ2v7X0I1ScyHuc5R0mydRPbjvVqT7M+Z0gIXWpErHuH/Wap8GtrkjwAm0hakCo2eEYKjimQDxJWi47wMVQhnkwX5QUyH2AJgLeMdfs45bfLtXQiNUvVRF6+nf1lolBtJS6zIFWz2L1wnVNJGF0yrExSS21nlhl6z1Ul6u68cSf391z/qUlBXt3x0lZGapwsgr3WoVuNdoQWfKSd5Ct6XzLnVlEKcJOye7KL6b4RQRbYAE4HgV4tlF4wW/ruiUhqtDEioOmg2dSahyZMZBYPNx4Jf2mgkQHr'}
2016-11-08T19:10:41.420 INFO:teuthology.kill:Traceback (most recent call last):
2016-11-08T19:10:41.420 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/nuke/__init__.py", line 281, in nuke_one
2016-11-08T19:10:41.420 INFO:teuthology.kill:    nuke_helper(ctx, should_unlock)
2016-11-08T19:10:41.420 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/nuke/__init__.py", line 324, in nuke_helper
2016-11-08T19:10:41.420 INFO:teuthology.kill:    remove_installed_packages(ctx)
2016-11-08T19:10:41.420 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/nuke/actions.py", line 318, in remove_installed_packages
2016-11-08T19:10:41.420 INFO:teuthology.kill:    rpm=rpms,
2016-11-08T19:10:41.420 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 537, in remove_packages
2016-11-08T19:10:41.420 INFO:teuthology.kill:    system_type], ctx, config, remote, pkgs[system_type])
2016-11-08T19:10:41.420 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 83, in __exit__
2016-11-08T19:10:41.421 INFO:teuthology.kill:    for result in self:
2016-11-08T19:10:41.421 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
2016-11-08T19:10:41.421 INFO:teuthology.kill:    resurrect_traceback(result)
2016-11-08T19:10:41.421 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
2016-11-08T19:10:41.421 INFO:teuthology.kill:    return func(*args, **kwargs)
2016-11-08T19:10:41.421 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 454, in _remove_deb
2016-11-08T19:10:41.421 INFO:teuthology.kill:    'autoremove',
2016-11-08T19:10:41.421 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 192, in run
2016-11-08T19:10:41.421 INFO:teuthology.kill:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2016-11-08T19:10:41.421 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 403, in run
2016-11-08T19:10:41.421 INFO:teuthology.kill:    r.wait()
2016-11-08T19:10:41.422 INFO:teuthology.kill:  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 166, in wait
2016-11-08T19:10:41.422 INFO:teuthology.kill:    label=self.label)
2016-11-08T19:10:41.422 INFO:teuthology.kill:CommandFailedError: Command failed on smithi057 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove'

History

#2 Updated by David Galloway 12 months ago

I kept smithi057 locked for post mortem.

#3 Updated by David Galloway 8 months ago

Here's another case of nuke failing: http://paste2.org/8jzbtLGa

Note that ceph-common is explicitly removed early before the failure:

2017-02-14 17:27:22,797.797 INFO:teuthology.task.install.deb:Removing packages: ceph, ceph-mds, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs1, libcephfs-java, libcephfs-jni, librados2, librbd1, rbd-fuse, ceph-dbg, ceph-mds-dbg, ceph-common-dbg, ceph-fuse-dbg, radosgw-dbg, libcephfs1-dbg, librados2-dbg, librbd1-dbg, salt-common, salt-minion, calamari-server, python-rados, multipath-tools on Debian system.

2017-02-14 17:27:22,797.797 INFO:teuthology.orchestra.run.mira056:Running: 'for d in ceph ceph-mds ceph-common ceph-fuse ceph-test radosgw python-ceph libcephfs1 libcephfs-java libcephfs-jni librados2 librbd1 rbd-fuse ceph-dbg ceph-mds-dbg ceph-common-dbg ceph-fuse-dbg radosgw-dbg libcephfs1-dbg librados2-dbg librbd1-dbg salt-common salt-minion calamari-server python-rados multipath-tools ; do sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" purge $d || true ; done'

And the apt-get autoremove gets hung up on ceph-common later.

CommandFailedError: Command failed on mira107 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove'
2017-02-14 17:27:38,270.270 INFO:teuthology.orchestra.run.mira056.stdout:Building dependency tree...
2017-02-14 17:27:38,271.271 INFO:teuthology.orchestra.run.mira056.stdout:Reading state information...
2017-02-14 17:27:38,392.392 INFO:teuthology.orchestra.run.mira056.stdout:You might want to run 'apt-get -f install' to correct these.
2017-02-14 17:27:38,392.392 INFO:teuthology.orchestra.run.mira056.stdout:The following packages have unmet dependencies:
2017-02-14 17:27:38,392.392 INFO:teuthology.orchestra.run.mira056.stdout: ceph-base : Depends: ceph-common (= 12.0.0-294-gd01fdc5-1trusty) but it is not installed
2017-02-14 17:27:38,397.397 INFO:teuthology.orchestra.run.mira056.stderr:E: Unmet dependencies. Try using -f.

Manually

root@mira056:~# apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove
Reading package lists... Done
Building dependency tree       
Reading state information... Done
You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
 ceph-base : Depends: ceph-common (= 12.0.0-294-gd01fdc5-1trusty) but it is not installed
E: Unmet dependencies. Try using -f.

# And adding '-f' *installs* ceph-common but doesn't purge anything.

root@mira056:~# apt-get -y -f --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
  ceph-common
Suggested packages:
  ceph-base ceph-mds
The following packages will be REMOVED:
  xmlstarlet
The following NEW packages will be installed:
  ceph-common
0 upgraded, 1 newly installed, 1 to remove and 177 not upgraded.
Need to get 0 B/9,531 kB of archives.
After this operation, 38.5 MB of additional disk space will be used.
(Reading database ... 100087 files and directories currently installed.)
Removing xmlstarlet (1.5.0-1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Selecting previously unselected package ceph-common.
(Reading database ... 99840 files and directories currently installed.)
Preparing to unpack .../ceph-common_12.0.0-294-gd01fdc5-1trusty_amd64.deb ...
Unpacking ceph-common (12.0.0-294-gd01fdc5-1trusty) ...
dpkg: warning: ceph-common: conffile 'etc/default/ceph' is not a plain file or symlink (= '/etc/default/ceph')
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up ceph-common (12.0.0-294-gd01fdc5-1trusty) ...
dpkg: warning: ceph-common: conffile '/etc/default/ceph' is not a plain file or symlink (= '/etc/default/ceph')
Setting system user ceph properties..usermod: no changes
..done
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for libc-bin (2.19-0ubuntu6.7) ...

Do we just want to add ceph-base to the list of packages to remove earlier in the process?

#4 Updated by Dan Mick 8 months ago

299b7d06ac18c5cd30b8b65c7d25df9fc00287db introduced a syntax error into ceph_common.sh that prevents at least some packages from uninstalling cleanly

#5 Updated by Dan Mick 8 months ago

  • Status changed from New to In Progress
  • Assignee set to Dan Mick
  • Priority changed from Normal to Urgent

#6 Updated by Dan Mick 8 months ago

  • Project changed from teuthology to Ceph

#7 Updated by Dan Mick 8 months ago

  • Subject changed from teuthology-nuke fails to remove packages with uninstalled dependencies to ceph-init syntax error breaks package remove, daemon stop

#8 Updated by Dan Mick 8 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF