Ceph » devops

Running teuthology-nuke --stale --owner scheduled_teuthology@teuthology --unlock would not complete on plana59:

2015-02-09 08:28:19,854.854 INFO:teuthology.orchestra.run.plana59.stderr:Errors were encountered while processing:
2015-02-09 08:28:19,854.854 INFO:teuthology.orchestra.run.plana59.stderr: librbd1
2015-02-09 08:28:19,855.855 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/ubuntu/teuthology/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/ubuntu/teuthology/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/ubuntu/teuthology/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/ubuntu/teuthology/teuthology/task/install.py", line 581, in _remove_deb
    'dpkg', '-P', '--force-remove-reinstreq',
  File "/home/ubuntu/teuthology/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/ubuntu/teuthology/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/ubuntu/teuthology/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on plana59 with status 123: "dpkg -l | grep '^.HR' | awk '{print $2}' | sudo xargs --no-run-if-empty dpkg -P --force-remove-reinstreq" 
2015-02-09 08:28:19,878.878 ERROR:teuthology.nuke:Could not nuke all targets in {u'plana59.front.sepia.ceph.com': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaHG83zRXo6ydv6IGWDFTf6YNjWG9M5LRbbYIpPXKOqCg9zfI/4ZjymLpznESFIACVrqe06jqD7uvsQPOlbcm3W/H44su70C21KrzMs77IpskMT7tYgCzY75uxbwg949qYIRf1SEY2RW0Bf2zldbOeKAY/TcnGIkLtc4NCIDPfCxMG0rAJJgUAwbvbKVUqLKe/jcyu3RiiAxV3TGjTAzTz+XHwT46gDXB5Fxt49Sfx+AgpILHk7DvN/HILtU3gRT9ac0D2WlQi1sJLDgjeTAZxyfpRR5iZH4tWYBFIS7C4ugHYye95zUYTc/3Jt364Jl/giUherGjE5od7p65VjxRJ'}
Traceback (most recent call last):
  File "/home/ubuntu/teuthology/teuthology/nuke.py", line 446, in nuke_one
    nuke_helper(ctx, should_unlock)
  File "/home/ubuntu/teuthology/teuthology/nuke.py", line 536, in nuke_helper
    remove_installed_packages(ctx)
  File "/home/ubuntu/teuthology/teuthology/nuke.py", line 264, in remove_installed_packages
    ['salt-common', 'salt-minion', 'calamari-server']})
  File "/home/ubuntu/teuthology/teuthology/task/install.py", line 650, in remove_packages
    system_type], ctx, config, remote, pkgs[system_type])
  File "/home/ubuntu/teuthology/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/ubuntu/teuthology/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/ubuntu/teuthology/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/ubuntu/teuthology/teuthology/task/install.py", line 581, in _remove_deb
    'dpkg', '-P', '--force-remove-reinstreq',
  File "/home/ubuntu/teuthology/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/ubuntu/teuthology/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/ubuntu/teuthology/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on plana59 with status 123: "dpkg -l | grep '^.HR' | awk '{print $2}' | sudo xargs --no-run-if-empty dpkg -P --force-remove-reinstreq" 

and running thso most like will fix the issue:

ubuntu@plana59:~$ sudo apt-get -f install 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
  librados2 librados2-dbg librbd1 python-ceph
The following packages will be upgraded:
  librados2 librados2-dbg librbd1 python-ceph
4 upgraded, 0 newly installed, 0 to remove and 189 not upgraded.
6 not fully installed or removed.
Need to get 0 B/30.3 MB of archives.
After this operation, 15.7 MB of additional disk space will be used.
Do you want to continue [Y/n]?