Actions
Bug #46503
closed"power on" timeout too short
% Done:
0%
Source:
Tags:
Backport:
octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
2020-07-11T03:23:35.118 INFO:teuthology.orchestra.console:Power on smithi033
2020-07-11T03:23:35.119 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2020-07-11T03:23:35.132 INFO:teuthology.orchestra.console:Power on smithi026
2020-07-11T03:23:35.133 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2020-07-11T03:23:35.146 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-07-11T03:23:35.160 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-07-11T03:23:39.176 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-07-11T03:23:39.190 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-07-11T03:23:39.406 INFO:teuthology.orchestra.console:Power on for smithi026 completed
2020-07-11T03:23:39.508 INFO:teuthology.provision.fog.smithi026:Waiting for deploy to finish
2020-07-11T03:23:44.294 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-07-11T03:23:49.410 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-07-11T03:23:54.528 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-07-11T03:23:55.641 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (5) after waiting for 20.0 seconds
2020-07-11T03:23:55.643 ERROR:teuthology.orchestra.console:Failed to power on smithi033
2020-07-11T03:23:55.643 INFO:teuthology.provision.fog.smithi033:Waiting for deploy to finish
2020-07-11T03:25:55.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-07-11T03:26:04.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-07-11T03:26:13.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-07-11T03:26:22.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-07-11T03:26:31.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-07-11T03:26:40.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-07-11T03:27:46.931 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-07-11T03:27:47.179 INFO:teuthology.orchestra.run:Running command with timeout 600
2020-07-11T03:27:47.181 INFO:teuthology.orchestra.run.smithi026:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2020-07-11T03:27:47.517 INFO:teuthology.provision.fog.smithi026:Node is ready
2020-07-11T03:27:47.518 INFO:teuthology.orchestra.run.smithi026:> hostname
2020-07-11T03:27:47.548 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026.front.sepia.ceph.com
2020-07-11T03:27:47.549 INFO:teuthology.orchestra.run.smithi026:> grep smithi026.front.sepia.ceph.com /etc/hosts
2020-07-11T03:27:47.618 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26 smithi026.front.sepia.ceph.com smithi026
2020-07-11T03:27:47.619 INFO:teuthology.orchestra.run.smithi026:> sudo hostname smithi026
2020-07-11T03:27:47.752 INFO:teuthology.orchestra.run.smithi026:> sudo sed -i -e 's/smithi026.front.sepia.ceph.com/smithi026/g' /etc/hosts
2020-07-11T03:27:47.786 INFO:teuthology.orchestra.run.smithi026:> sudo sed -i -e 's/172.21.15.26/172.21.15.26/g' /etc/hosts
2020-07-11T03:27:47.859 INFO:teuthology.orchestra.run.smithi026:> sudo sed -i -e 's/smithi026.front.sepia.ceph.com/smithi026/g' /etc/hostname
2020-07-11T03:27:47.931 INFO:teuthology.orchestra.run.smithi026:> sudo hostname smithi026
2020-07-11T03:27:48.002 INFO:teuthology.provision.fog.smithi026:Deploy complete!
2020-07-11T03:27:48.003 DEBUG:teuthology.parallel:result is None
2020-07-11T03:38:46.722 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 93, in run_tasks
manager.__enter__()
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/lock_machines.py", line 78, in lock_machines
os_version, arch)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/lock/ops.py", line 145, in lock_many
reimaged[machine] = machines[machine]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/provision/__init__.py", line 39, in reimage
return obj.create()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/provision/fog.py", line 86, in create
self.wait_for_deploy_task(task_id)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/provision/fog.py", line 244, in wait_for_deploy_task
while proceed():
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 133, in __call__
raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (60) after waiting for 900 seconds
From: /ceph/teuthology-archive/pdonnell-2020-07-11_02:43:08-fs-wip-pdonnell-testing-20200711.001802-distro-basic-smithi/5214092/teuthology.log
Updated by Patrick Donnelly almost 4 years ago
- Status changed from In Progress to Fix Under Review
Updated by Patrick Donnelly almost 4 years ago
- Status changed from Fix Under Review to Resolved
Updated by Deepika Upadhyay over 3 years ago
- Backport set to octopus
seems like we would need octopus backport
/a/yuriw-2020-10-20_15:30:01-rados-wip-yuri5-testing-2020-10-07-1021-octopus-distro-basic-smithi/5542545/teuthology.log
Updated by Patrick Donnelly over 3 years ago
Deepika Upadhyay wrote:
seems like we would need octopus backport
/a/yuriw-2020-10-20_15:30:01-rados-wip-yuri5-testing-2020-10-07-1021-octopus-distro-basic-smithi/5542545/teuthology.log
This is a teuthology bug so it's already in use for octopus.
Actions