Project

General

Profile

Bug #36597

HTTPError: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/309/task

Added by Patrick Donnelly over 5 years ago. Updated over 5 years ago.

Status:
New
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

2018-10-25T03:34:21.362 INFO:teuthology.provision.fog.smithi029:Scheduling deploy of ubuntu 16.04
2018-10-25T03:34:21.367 INFO:teuthology.provision.fog.smithi193:Scheduling deploy of ubuntu 16.04
2018-10-25T03:34:21.395 INFO:teuthology.provision.fog.smithi184:Scheduling deploy of ubuntu 16.04
2018-10-25T03:34:21.649 ERROR:teuthology.provision.fog.smithi193:500: { "error": "Host is already a member of an active task" 
}
2018-10-25T03:34:21.650 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/run_tasks.py", line 89, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/task/internal/lock_machines.py", line 79, in lock_machines
    os_version, arch)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/lock/ops.py", line 125, in lock_many
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/parallel.py", line 85, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/parallel.py", line 99, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/parallel.py", line 22, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/provision/__init__.py", line 26, in reimage
    return fog_obj.create()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/provision/fog.py", line 78, in create
    task_id = self.schedule_deploy_task(host_id)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/provision/fog.py", line 200, in schedule_deploy_task
    data='{"taskTypeID": %i}' % deploy_id,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/provision/fog.py", line 122, in do_request
    resp.raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/virtualenv/local/lib/python2.7/site-packages/requests/models.py", line 893, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/309/task
2018-10-25T03:34:21.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.lock_machines
2018-10-25T03:34:21.660 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: HTTPError: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/309/task
2018-10-25T03:34:21.664 INFO:teuthology.run:Summary data:
{description: 'fs/upgrade/featureful_client/old_client/{bluestore.yaml clusters/1-mds-2-client-micro.yaml
    conf/{client.yaml mds.yaml mon.yaml osd.yaml} overrides/{frag_enable.yaml multimds/yes.yaml
    whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/{0-luminous.yaml
    1-client.yaml 2-upgrade.yaml 3-compat_client/no.yaml}}', failure_reason: '500
    Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/309/task',
  owner: scheduled_pdonnell@teuthology, status: dead, success: false}

2018-10-25T03:34:21.664 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/
2018-10-25T03:34:21.700 INFO:teuthology.orchestra.console:Power off smithi029
2018-10-25T03:34:21.700 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2018-10-25T03:34:21.729 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2018-10-25T03:34:21.741 INFO:teuthology.run:DEAD

/ceph/teuthology-archive/pdonnell-2018-10-24_23:40:49-fs-wip-pdonnell-testing-20181024.213729-distro-basic-smithi/3180722/teuthology.log


Related issues

Related to sepia - Bug #54389: Dead jobs caused by "Error reimaging machines: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/137/task" New

History

#1 Updated by Patrick Donnelly over 5 years ago

Another weird issue involving smithi193:

2018-10-25T03:34:21.230 INFO:teuthology.provision.fog.smithi193:Scheduling deploy of rhel 7.5
...
018-10-25T03:39:40.296 INFO:teuthology.orchestra.run.smithi193:Running: 'uname -m'
2018-10-25T03:39:40.303 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64
2018-10-25T03:39:40.303 INFO:teuthology.orchestra.run.smithi193:Running: 'cat /etc/os-release'
2018-10-25T03:39:40.375 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="Ubuntu" 
2018-10-25T03:39:40.375 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="16.04.4 LTS (Xenial Xerus)" 
2018-10-25T03:39:40.375 INFO:teuthology.orchestra.run.smithi193.stdout:ID=ubuntu
2018-10-25T03:39:40.376 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE=debian
2018-10-25T03:39:40.376 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="Ubuntu 16.04.4 LTS" 
2018-10-25T03:39:40.376 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="16.04" 
2018-10-25T03:39:40.376 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="http://www.ubuntu.com/" 
2018-10-25T03:39:40.376 INFO:teuthology.orchestra.run.smithi193.stdout:SUPPORT_URL="http://help.ubuntu.com/" 
2018-10-25T03:39:40.376 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 
2018-10-25T03:39:40.376 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_CODENAME=xenial
2018-10-25T03:39:40.377 INFO:teuthology.orchestra.run.smithi193.stdout:UBUNTU_CODENAME=xenial
2018-10-25T03:39:40.377 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server

For some reason, rhel was not deployed.

#2 Updated by Laura Flores about 2 years ago

  • Related to Bug #54389: Dead jobs caused by "Error reimaging machines: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/137/task" added

Also available in: Atom PDF