Actions
Bug #46503
closed"power on" timeout too short
% Done:
0%
Source:
Tags:
Backport:
octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
2020-07-11T03:23:35.118 INFO:teuthology.orchestra.console:Power on smithi033 2020-07-11T03:23:35.119 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-07-11T03:23:35.132 INFO:teuthology.orchestra.console:Power on smithi026 2020-07-11T03:23:35.133 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-07-11T03:23:35.146 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-11T03:23:35.160 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-11T03:23:39.176 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-11T03:23:39.190 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-11T03:23:39.406 INFO:teuthology.orchestra.console:Power on for smithi026 completed 2020-07-11T03:23:39.508 INFO:teuthology.provision.fog.smithi026:Waiting for deploy to finish 2020-07-11T03:23:44.294 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-11T03:23:49.410 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-11T03:23:54.528 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-07-11T03:23:55.641 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (5) after waiting for 20.0 seconds 2020-07-11T03:23:55.643 ERROR:teuthology.orchestra.console:Failed to power on smithi033 2020-07-11T03:23:55.643 INFO:teuthology.provision.fog.smithi033:Waiting for deploy to finish 2020-07-11T03:25:55.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-11T03:26:04.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-11T03:26:13.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-11T03:26:22.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-11T03:26:31.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-11T03:26:40.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-11T03:27:46.931 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-07-11T03:27:47.179 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-07-11T03:27:47.181 INFO:teuthology.orchestra.run.smithi026:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-07-11T03:27:47.517 INFO:teuthology.provision.fog.smithi026:Node is ready 2020-07-11T03:27:47.518 INFO:teuthology.orchestra.run.smithi026:> hostname 2020-07-11T03:27:47.548 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026.front.sepia.ceph.com 2020-07-11T03:27:47.549 INFO:teuthology.orchestra.run.smithi026:> grep smithi026.front.sepia.ceph.com /etc/hosts 2020-07-11T03:27:47.618 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26 smithi026.front.sepia.ceph.com smithi026 2020-07-11T03:27:47.619 INFO:teuthology.orchestra.run.smithi026:> sudo hostname smithi026 2020-07-11T03:27:47.752 INFO:teuthology.orchestra.run.smithi026:> sudo sed -i -e 's/smithi026.front.sepia.ceph.com/smithi026/g' /etc/hosts 2020-07-11T03:27:47.786 INFO:teuthology.orchestra.run.smithi026:> sudo sed -i -e 's/172.21.15.26/172.21.15.26/g' /etc/hosts 2020-07-11T03:27:47.859 INFO:teuthology.orchestra.run.smithi026:> sudo sed -i -e 's/smithi026.front.sepia.ceph.com/smithi026/g' /etc/hostname 2020-07-11T03:27:47.931 INFO:teuthology.orchestra.run.smithi026:> sudo hostname smithi026 2020-07-11T03:27:48.002 INFO:teuthology.provision.fog.smithi026:Deploy complete! 2020-07-11T03:27:48.003 DEBUG:teuthology.parallel:result is None 2020-07-11T03:38:46.722 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 93, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/lock_machines.py", line 78, in lock_machines os_version, arch) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/lock/ops.py", line 145, in lock_many reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/provision/fog.py", line 244, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (60) after waiting for 900 seconds
From: /ceph/teuthology-archive/pdonnell-2020-07-11_02:43:08-fs-wip-pdonnell-testing-20200711.001802-distro-basic-smithi/5214092/teuthology.log
Actions