Project

General

Profile

Bug #10803

machines would not nuke, require apt-get -f manual run (e.g. librados2 librados2-dbg librbd1 python-ceph)

Added by Yuri Weinstein about 9 years ago. Updated almost 9 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Running teuthology-nuke --stale --owner scheduled_teuthology@teuthology --unlock would not complete on plana59:

2015-02-09 08:28:19,854.854 INFO:teuthology.orchestra.run.plana59.stderr:Errors were encountered while processing:
2015-02-09 08:28:19,854.854 INFO:teuthology.orchestra.run.plana59.stderr: librbd1
2015-02-09 08:28:19,855.855 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/ubuntu/teuthology/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/ubuntu/teuthology/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/ubuntu/teuthology/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/ubuntu/teuthology/teuthology/task/install.py", line 581, in _remove_deb
    'dpkg', '-P', '--force-remove-reinstreq',
  File "/home/ubuntu/teuthology/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/ubuntu/teuthology/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/ubuntu/teuthology/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on plana59 with status 123: "dpkg -l | grep '^.HR' | awk '{print $2}' | sudo xargs --no-run-if-empty dpkg -P --force-remove-reinstreq" 
2015-02-09 08:28:19,878.878 ERROR:teuthology.nuke:Could not nuke all targets in {u'plana59.front.sepia.ceph.com': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaHG83zRXo6ydv6IGWDFTf6YNjWG9M5LRbbYIpPXKOqCg9zfI/4ZjymLpznESFIACVrqe06jqD7uvsQPOlbcm3W/H44su70C21KrzMs77IpskMT7tYgCzY75uxbwg949qYIRf1SEY2RW0Bf2zldbOeKAY/TcnGIkLtc4NCIDPfCxMG0rAJJgUAwbvbKVUqLKe/jcyu3RiiAxV3TGjTAzTz+XHwT46gDXB5Fxt49Sfx+AgpILHk7DvN/HILtU3gRT9ac0D2WlQi1sJLDgjeTAZxyfpRR5iZH4tWYBFIS7C4ugHYye95zUYTc/3Jt364Jl/giUherGjE5od7p65VjxRJ'}
Traceback (most recent call last):
  File "/home/ubuntu/teuthology/teuthology/nuke.py", line 446, in nuke_one
    nuke_helper(ctx, should_unlock)
  File "/home/ubuntu/teuthology/teuthology/nuke.py", line 536, in nuke_helper
    remove_installed_packages(ctx)
  File "/home/ubuntu/teuthology/teuthology/nuke.py", line 264, in remove_installed_packages
    ['salt-common', 'salt-minion', 'calamari-server']})
  File "/home/ubuntu/teuthology/teuthology/task/install.py", line 650, in remove_packages
    system_type], ctx, config, remote, pkgs[system_type])
  File "/home/ubuntu/teuthology/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/ubuntu/teuthology/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/ubuntu/teuthology/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/ubuntu/teuthology/teuthology/task/install.py", line 581, in _remove_deb
    'dpkg', '-P', '--force-remove-reinstreq',
  File "/home/ubuntu/teuthology/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/ubuntu/teuthology/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/ubuntu/teuthology/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on plana59 with status 123: "dpkg -l | grep '^.HR' | awk '{print $2}' | sudo xargs --no-run-if-empty dpkg -P --force-remove-reinstreq" 

and running thso most like will fix the issue:

ubuntu@plana59:~$ sudo apt-get -f install 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
  librados2 librados2-dbg librbd1 python-ceph
The following packages will be upgraded:
  librados2 librados2-dbg librbd1 python-ceph
4 upgraded, 0 newly installed, 0 to remove and 189 not upgraded.
6 not fully installed or removed.
Need to get 0 B/30.3 MB of archives.
After this operation, 15.7 MB of additional disk space will be used.
Do you want to continue [Y/n]? 

History

#1 Updated by Zack Cerza about 9 years ago

How does apt-get -f install fix the issue? Doesn't that still leave ceph packages behind?

#2 Updated by Zack Cerza about 9 years ago

Also I think we need more output to actually diagnose the problem. The traceback isn't enough when a command fails; we need the actual error from the command output.

#3 Updated by Yuri Weinstein about 9 years ago

  • Description updated (diff)

#4 Updated by Yuri Weinstein about 9 years ago

Added more traceback info above.

#5 Updated by Zack Cerza about 9 years ago

As I said, the traceback is not enough. We need to know why the command failed, so we need its output.

#6 Updated by Yuri Weinstein about 9 years ago

Zack, here it is:

2015-02-09 08:28:17,737.737 INFO:teuthology.orchestra.run.plana59.stdout:Reading package lists...
2015-02-09 08:28:18,056.056 INFO:teuthology.orchestra.run.plana59.stdout:Building dependency tree...
2015-02-09 08:28:18,058.058 INFO:teuthology.orchestra.run.plana59.stdout:Reading state information...
2015-02-09 08:28:18,151.151 INFO:teuthology.orchestra.run.plana59.stdout:You might want to run 'apt-get -f install' to correct these:
2015-02-09 08:28:18,151.151 INFO:teuthology.orchestra.run.plana59.stdout:The following packages have unmet dependencies:
2015-02-09 08:28:18,152.152 INFO:teuthology.orchestra.run.plana59.stdout: ceph-common : Depends: python-ceph (= 0.92-920-ga08d3be-1precise) but 0.80.8-22-g26966c4-1precise is to be installed
2015-02-09 08:28:18,152.152 INFO:teuthology.orchestra.run.plana59.stdout: librbd1 : Depends: librados2 (= 0.92-920-ga08d3be-1precise) but 0.80.8-22-g26966c4-1precise is to be installed
2015-02-09 08:28:18,172.172 INFO:teuthology.orchestra.run.plana59.stderr:E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
2015-02-09 08:28:18,186.186 INFO:teuthology.orchestra.run.plana59.stdout:Reading package lists...
2015-02-09 08:28:18,489.489 INFO:teuthology.orchestra.run.plana59.stdout:Building dependency tree...
2015-02-09 08:28:18,491.491 INFO:teuthology.orchestra.run.plana59.stdout:Reading state information...
2015-02-09 08:28:18,577.577 INFO:teuthology.orchestra.run.plana59.stderr:E: Unable to locate package salt-common
2015-02-09 08:28:18,589.589 INFO:teuthology.orchestra.run.plana59.stdout:Reading package lists...
2015-02-09 08:28:18,903.903 INFO:teuthology.orchestra.run.plana59.stdout:Building dependency tree...
2015-02-09 08:28:18,905.905 INFO:teuthology.orchestra.run.plana59.stdout:Reading state information...
2015-02-09 08:28:19,025.025 INFO:teuthology.orchestra.run.plana59.stderr:E: Unable to locate package salt-minion
2015-02-09 08:28:19,042.042 INFO:teuthology.orchestra.run.plana59.stdout:Reading package lists...
2015-02-09 08:28:19,437.437 INFO:teuthology.orchestra.run.plana59.stdout:Building dependency tree...
2015-02-09 08:28:19,439.439 INFO:teuthology.orchestra.run.plana59.stdout:Reading state information...
2015-02-09 08:28:19,559.559 INFO:teuthology.orchestra.run.plana59.stderr:E: Unable to locate package calamari-server
2015-02-09 08:28:19,561.561 INFO:teuthology.orchestra.run.plana59:Running: "dpkg -l | grep '^.HR' | awk '{print $2}' | sudo xargs --no-run-if-empty dpkg -P --force-remove-reinstreq" 
2015-02-09 08:28:19,753.753 INFO:teuthology.orchestra.run.plana59.stderr:dpkg: dependency problems prevent removal of librbd1:
2015-02-09 08:28:19,753.753 INFO:teuthology.orchestra.run.plana59.stderr: ceph-test depends on librbd1; however:
2015-02-09 08:28:19,754.754 INFO:teuthology.orchestra.run.plana59.stderr:  Package librbd1 is to be removed.
2015-02-09 08:28:19,754.754 INFO:teuthology.orchestra.run.plana59.stderr: qemu-kvm depends on librbd1; however:
2015-02-09 08:28:19,754.754 INFO:teuthology.orchestra.run.plana59.stderr:  Package librbd1 is to be removed.
2015-02-09 08:28:19,754.754 INFO:teuthology.orchestra.run.plana59.stderr: qemu-utils depends on librbd1.
2015-02-09 08:28:19,754.754 INFO:teuthology.orchestra.run.plana59.stderr: python-ceph depends on librbd1.
2015-02-09 08:28:19,755.755 INFO:teuthology.orchestra.run.plana59.stderr: tgt depends on librbd1.
2015-02-09 08:28:19,755.755 INFO:teuthology.orchestra.run.plana59.stderr:dpkg: error processing librbd1 (--purge):
2015-02-09 08:28:19,755.755 INFO:teuthology.orchestra.run.plana59.stderr: dependency problems - not removing

#7 Updated by Zack Cerza about 9 years ago

Thanks!

#8 Updated by Andrew Schoen almost 9 years ago

  • Assignee changed from Sandon Van Ness to Andrew Schoen

Also available in: Atom PDF