Bug #17826
ceph-init syntax error breaks package remove, daemon stop
% Done:
0%
Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2016-11-08T19:10:41.412 INFO:teuthology.kill:2016-11-08 19:10:41,412.412 INFO:teuthology.orchestra.run.smithi057.stdout:You might want to run 'apt-get -f install' to correct these. 2016-11-08T19:10:41.412 INFO:teuthology.kill:2016-11-08 19:10:41,412.412 INFO:teuthology.orchestra.run.smithi057.stdout:The following packages have unmet dependencies: 2016-11-08T19:10:41.413 INFO:teuthology.kill:2016-11-08 19:10:41,412.412 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr : Depends: ceph-base (= 11.0.2-1289-g036e761-1trusty) but it is not installed 2016-11-08T19:10:41.416 INFO:teuthology.kill:2016-11-08 19:10:41,415.415 INFO:teuthology.orchestra.run.smithi057.stderr:E: Unmet dependencies. Try using -f. 2016-11-08T19:10:41.417 INFO:teuthology.kill:2016-11-08 19:10:41,417.417 ERROR:teuthology.parallel:Exception in parallel execution 2016-11-08T19:10:41.418 INFO:teuthology.kill:Traceback (most recent call last): 2016-11-08T19:10:41.418 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 83, in __exit__ 2016-11-08T19:10:41.418 INFO:teuthology.kill: for result in self: 2016-11-08T19:10:41.418 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next 2016-11-08T19:10:41.418 INFO:teuthology.kill: resurrect_traceback(result) 2016-11-08T19:10:41.418 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback 2016-11-08T19:10:41.418 INFO:teuthology.kill: return func(*args, **kwargs) 2016-11-08T19:10:41.418 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 454, in _remove_deb 2016-11-08T19:10:41.418 INFO:teuthology.kill: 'autoremove', 2016-11-08T19:10:41.419 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 192, in run 2016-11-08T19:10:41.419 INFO:teuthology.kill: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2016-11-08T19:10:41.419 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 403, in run 2016-11-08T19:10:41.419 INFO:teuthology.kill: r.wait() 2016-11-08T19:10:41.419 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 166, in wait 2016-11-08T19:10:41.419 INFO:teuthology.kill: label=self.label) 2016-11-08T19:10:41.419 INFO:teuthology.kill:CommandFailedError: Command failed on smithi057 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove' 2016-11-08T19:10:41.419 INFO:teuthology.kill:2016-11-08 19:10:41,417.417 ERROR:teuthology.nuke:Could not nuke {'ubuntu@smithi057.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD4MKWct6JS0SIfEJOJopEb+e9kvl57af0a8LKuckHUsF0wA5AH9q5hUZ2v7X0I1ScyHuc5R0mydRPbjvVqT7M+Z0gIXWpErHuH/Wap8GtrkjwAm0hakCo2eEYKjimQDxJWi47wMVQhnkwX5QUyH2AJgLeMdfs45bfLtXQiNUvVRF6+nf1lolBtJS6zIFWz2L1wnVNJGF0yrExSS21nlhl6z1Ul6u68cSf391z/qUlBXt3x0lZGapwsgr3WoVuNdoQWfKSd5Ct6XzLnVlEKcJOye7KL6b4RQRbYAE4HgV4tlF4wW/ruiUhqtDEioOmg2dSahyZMZBYPNx4Jf2mgkQHr'} 2016-11-08T19:10:41.420 INFO:teuthology.kill:Traceback (most recent call last): 2016-11-08T19:10:41.420 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/nuke/__init__.py", line 281, in nuke_one 2016-11-08T19:10:41.420 INFO:teuthology.kill: nuke_helper(ctx, should_unlock) 2016-11-08T19:10:41.420 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/nuke/__init__.py", line 324, in nuke_helper 2016-11-08T19:10:41.420 INFO:teuthology.kill: remove_installed_packages(ctx) 2016-11-08T19:10:41.420 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/nuke/actions.py", line 318, in remove_installed_packages 2016-11-08T19:10:41.420 INFO:teuthology.kill: rpm=rpms, 2016-11-08T19:10:41.420 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 537, in remove_packages 2016-11-08T19:10:41.420 INFO:teuthology.kill: system_type], ctx, config, remote, pkgs[system_type]) 2016-11-08T19:10:41.420 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 83, in __exit__ 2016-11-08T19:10:41.421 INFO:teuthology.kill: for result in self: 2016-11-08T19:10:41.421 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next 2016-11-08T19:10:41.421 INFO:teuthology.kill: resurrect_traceback(result) 2016-11-08T19:10:41.421 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback 2016-11-08T19:10:41.421 INFO:teuthology.kill: return func(*args, **kwargs) 2016-11-08T19:10:41.421 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 454, in _remove_deb 2016-11-08T19:10:41.421 INFO:teuthology.kill: 'autoremove', 2016-11-08T19:10:41.421 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 192, in run 2016-11-08T19:10:41.421 INFO:teuthology.kill: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2016-11-08T19:10:41.421 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 403, in run 2016-11-08T19:10:41.421 INFO:teuthology.kill: r.wait() 2016-11-08T19:10:41.422 INFO:teuthology.kill: File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 166, in wait 2016-11-08T19:10:41.422 INFO:teuthology.kill: label=self.label) 2016-11-08T19:10:41.422 INFO:teuthology.kill:CommandFailedError: Command failed on smithi057 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove'
History
#1 Updated by David Galloway over 7 years ago
#2 Updated by David Galloway over 7 years ago
I kept smithi057 locked for post mortem.
#3 Updated by David Galloway about 7 years ago
Here's another case of nuke failing: http://paste2.org/8jzbtLGa
Note that ceph-common is explicitly removed early before the failure:
2017-02-14 17:27:22,797.797 INFO:teuthology.task.install.deb:Removing packages: ceph, ceph-mds, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs1, libcephfs-java, libcephfs-jni, librados2, librbd1, rbd-fuse, ceph-dbg, ceph-mds-dbg, ceph-common-dbg, ceph-fuse-dbg, radosgw-dbg, libcephfs1-dbg, librados2-dbg, librbd1-dbg, salt-common, salt-minion, calamari-server, python-rados, multipath-tools on Debian system. 2017-02-14 17:27:22,797.797 INFO:teuthology.orchestra.run.mira056:Running: 'for d in ceph ceph-mds ceph-common ceph-fuse ceph-test radosgw python-ceph libcephfs1 libcephfs-java libcephfs-jni librados2 librbd1 rbd-fuse ceph-dbg ceph-mds-dbg ceph-common-dbg ceph-fuse-dbg radosgw-dbg libcephfs1-dbg librados2-dbg librbd1-dbg salt-common salt-minion calamari-server python-rados multipath-tools ; do sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" purge $d || true ; done'
And the apt-get autoremove
gets hung up on ceph-common later.
CommandFailedError: Command failed on mira107 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove' 2017-02-14 17:27:38,270.270 INFO:teuthology.orchestra.run.mira056.stdout:Building dependency tree... 2017-02-14 17:27:38,271.271 INFO:teuthology.orchestra.run.mira056.stdout:Reading state information... 2017-02-14 17:27:38,392.392 INFO:teuthology.orchestra.run.mira056.stdout:You might want to run 'apt-get -f install' to correct these. 2017-02-14 17:27:38,392.392 INFO:teuthology.orchestra.run.mira056.stdout:The following packages have unmet dependencies: 2017-02-14 17:27:38,392.392 INFO:teuthology.orchestra.run.mira056.stdout: ceph-base : Depends: ceph-common (= 12.0.0-294-gd01fdc5-1trusty) but it is not installed 2017-02-14 17:27:38,397.397 INFO:teuthology.orchestra.run.mira056.stderr:E: Unmet dependencies. Try using -f.
Manually
root@mira056:~# apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove Reading package lists... Done Building dependency tree Reading state information... Done You might want to run 'apt-get -f install' to correct these. The following packages have unmet dependencies: ceph-base : Depends: ceph-common (= 12.0.0-294-gd01fdc5-1trusty) but it is not installed E: Unmet dependencies. Try using -f. # And adding '-f' *installs* ceph-common but doesn't purge anything. root@mira056:~# apt-get -y -f --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" autoremove Reading package lists... Done Building dependency tree Reading state information... Done Correcting dependencies... Done The following extra packages will be installed: ceph-common Suggested packages: ceph-base ceph-mds The following packages will be REMOVED: xmlstarlet The following NEW packages will be installed: ceph-common 0 upgraded, 1 newly installed, 1 to remove and 177 not upgraded. Need to get 0 B/9,531 kB of archives. After this operation, 38.5 MB of additional disk space will be used. (Reading database ... 100087 files and directories currently installed.) Removing xmlstarlet (1.5.0-1) ... Processing triggers for man-db (2.6.7.1-1ubuntu1) ... Selecting previously unselected package ceph-common. (Reading database ... 99840 files and directories currently installed.) Preparing to unpack .../ceph-common_12.0.0-294-gd01fdc5-1trusty_amd64.deb ... Unpacking ceph-common (12.0.0-294-gd01fdc5-1trusty) ... dpkg: warning: ceph-common: conffile 'etc/default/ceph' is not a plain file or symlink (= '/etc/default/ceph') Processing triggers for ureadahead (0.100.0-16) ... Processing triggers for man-db (2.6.7.1-1ubuntu1) ... Setting up ceph-common (12.0.0-294-gd01fdc5-1trusty) ... dpkg: warning: ceph-common: conffile '/etc/default/ceph' is not a plain file or symlink (= '/etc/default/ceph') Setting system user ceph properties..usermod: no changes ..done Processing triggers for ureadahead (0.100.0-16) ... Processing triggers for libc-bin (2.19-0ubuntu6.7) ...
Do we just want to add ceph-base to the list of packages to remove earlier in the process?
#4 Updated by Dan Mick about 7 years ago
299b7d06ac18c5cd30b8b65c7d25df9fc00287db introduced a syntax error into ceph_common.sh that prevents at least some packages from uninstalling cleanly
#5 Updated by Dan Mick about 7 years ago
- Status changed from New to In Progress
- Assignee set to Dan Mick
- Priority changed from Normal to Urgent
#6 Updated by Dan Mick about 7 years ago
- Project changed from teuthology to Ceph
#7 Updated by Dan Mick about 7 years ago
- Subject changed from teuthology-nuke fails to remove packages with uninstalled dependencies to ceph-init syntax error breaks package remove, daemon stop
#8 Updated by Dan Mick about 7 years ago
- Status changed from In Progress to Resolved