Infrastructure » sepia

I see more examples like this during nuke, power cycle seems helping solve the issue.

  yuriw@teuthology ~ [23:17:28]> for owner in $(teuthology-lock --summary | grep -i scheduled | awk '{ print $4 }' | sort -u); do echo $owner; echo "========"; ~/temp/teuthology/virtualenv/bin/teuthology-nuke --stale --owner $owner --unlock; echo "======== "; done
....
======== 
scheduled_kchai@teuthology
========
2017-02-27 23:17:34,843.843 INFO:teuthology.nuke:targets:
  mira102.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeeHy1slAQa7p2E6PFkgbBgOFF+PQnFLfdkIom7BelXK81g4wlSVP7wuh4Zan69gBca1Z4njAzWg6AS+eocRuZzlwFXOVOFdDyUvh0wNxx+BbQRGS4acs2sbgAtHSPbsEUyLqvz5R4Z3KGlMAQEaQSn+dkbqG8bd7I6iKEs7nDue0StnxdvXE+/BjURvp49MZQLDDwDlP8zOi5sEtTeyWahwFzki3GCFfhg0mIpilyoRxMM94Cea29vYaW4j9WHctvuKp+fJwobLr1Th222JO+QwvGIjYEfc2e+Pd8/wf0HjNG2qtl/Nt7T8gaJRovjOvS8/n7494tnv0QJ2QDZgvX
2017-02-27 23:17:34,864.864 INFO:teuthology.task.internal.check_lock:Checking locks...
2017-02-27 23:17:34,925.925 INFO:teuthology.nuke.actions:checking console status of mira102.ipmi.sepia.ceph.com
2017-02-27 23:17:35,366.366 INFO:teuthology.nuke.actions:console ready on mira102.ipmi.sepia.ceph.com
2017-02-27 23:17:35,366.366 INFO:teuthology.task.internal:Opening connections...
2017-02-27 23:17:35,737.737 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules...
2017-02-27 23:17:35,738.738 INFO:teuthology.orchestra.run.mira102:Running: "sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'" 
2017-02-27 23:17:35,830.830 INFO:teuthology.orchestra.run.mira102.stderr:sh: 1: iptables-save: Input/output error
2017-02-27 23:17:35,830.830 INFO:teuthology.orchestra.run.mira102.stderr:sh: 1: iptables-restore: Input/output error
2017-02-27 23:17:35,832.832 ERROR:teuthology.nuke:Could not nuke {u'mira102.front.sepia.ceph.com': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeeHy1slAQa7p2E6PFkgbBgOFF+PQnFLfdkIom7BelXK81g4wlSVP7wuh4Zan69gBca1Z4njAzWg6AS+eocRuZzlwFXOVOFdDyUvh0wNxx+BbQRGS4acs2sbgAtHSPbsEUyLqvz5R4Z3KGlMAQEaQSn+dkbqG8bd7I6iKEs7nDue0StnxdvXE+/BjURvp49MZQLDDwDlP8zOi5sEtTeyWahwFzki3GCFfhg0mIpilyoRxMM94Cea29vYaW4j9WHctvuKp+fJwobLr1Th222JO+QwvGIjYEfc2e+Pd8/wf0HjNG2qtl/Nt7T8gaJRovjOvS8/n7494tnv0QJ2QDZgvX'}
Traceback (most recent call last):
  File "/home/yuriw/temp/teuthology/teuthology/nuke/__init__.py", line 280, in nuke_one
    nuke_helper(ctx, should_unlock)
  File "/home/yuriw/temp/teuthology/teuthology/nuke/__init__.py", line 315, in nuke_helper
    clear_firewall(ctx)
  File "/home/yuriw/temp/teuthology/teuthology/nuke/actions.py", line 23, in clear_firewall
    "iptables-save | grep -v teuthology | iptables-restore" 
  File "/home/yuriw/temp/teuthology/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/yuriw/temp/teuthology/teuthology/orchestra/remote.py", line 193, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/yuriw/temp/teuthology/teuthology/orchestra/run.py", line 403, in run
    r.wait()
  File "/home/yuriw/temp/teuthology/teuthology/orchestra/run.py", line 166, in wait
    label=self.label)
CommandFailedError: Command failed on mira102 with status 2: "sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'" 
2017-02-27 23:17:35,836.836 ERROR:teuthology.nuke:Could not nuke the following targets:
targets:
  mira102.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeeHy1slAQa7p2E6PFkgbBgOFF+PQnFLfdkIom7BelXK81g4wlSVP7wuh4Zan69gBca1Z4njAzWg6AS+eocRuZzlwFXOVOFdDyUvh0wNxx+BbQRGS4acs2sbgAtHSPbsEUyLqvz5R4Z3KGlMAQEaQSn+dkbqG8bd7I6iKEs7nDue0StnxdvXE+/BjURvp49MZQLDDwDlP8zOi5sEtTeyWahwFzki3GCFfhg0mIpilyoRxMM94Cea29vYaW4j9WHctvuKp+fJwobLr1Th222JO+QwvGIjYEfc2e+Pd8/wf0HjNG2qtl/Nt7T8gaJRovjOvS8/n7494tnv0QJ2QDZgvX