Tools » paddles

swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/5950060 killed a dispatcher.

Job Yaml¶

archive_path: /home/teuthworker/archive/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/5950060
branch: wip-swagner2-testing-2021-03-09-1019-pacific
description: rados:cephadm/smoke/{distro/ubuntu_20.04_kubic_testing fixed-2 mon_election/connectivity
  start}
email: swagner@suse.com
first_in_suite: false
job_id: '5950060'
kernel:
  kdb: true
  sha1: distro
last_in_suite: false
machine_type: smithi
name: swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi
nuke-on-error: true
openstack:
- volumes:
    count: 4
    size: 10
os_type: ubuntu
os_version: '20.04'
overrides:
  admin_socket:
    branch: wip-swagner2-testing-2021-03-09-1019-pacific
  ceph:
    conf:
      global:
        mon election default strategy: 3
      mgr:
        debug mgr: 20
        debug ms: 1
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 25
        osd shutdown pgref assert: true
    log-ignorelist:
    - \(MDS_ALL_DOWN\)
    - \(MDS_UP_LESS_THAN_MAX\)
    log-whitelist:
    - \(MDS_ALL_DOWN\)
    - \(MDS_UP_LESS_THAN_MAX\)
    sha1: b66e292cfec45247a651159e65386f496cb4d718
  ceph-deploy:
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        osd default pool size: 2
  install:
    ceph:
      sha1: b66e292cfec45247a651159e65386f496cb4d718
  workunit:
    branch: pacific-backport-39495-39630-39536-39537-39415-39421-39420-39474-39562-39613-
    sha1: 732027a3aaa7f0273320259c12e605cddca32d95
owner: scheduled_swagner@teuthology
priority: 75
repo: git://git.ceph.com/ceph-ci.git
roles:
- - mon.a
  - mon.c
  - mgr.y
  - osd.0
  - osd.1
  - osd.2
  - osd.3
  - client.0
  - ceph.rgw.realm.zone.a
  - node-exporter.a
  - alertmanager.a
- - mon.b
  - mgr.x
  - osd.4
  - osd.5
  - osd.6
  - osd.7
  - client.1
  - prometheus.a
  - grafana.a
  - node-exporter.b
  - ceph.iscsi.iscsi.a
sha1: b66e292cfec45247a651159e65386f496cb4d718
sleep_before_teardown: 0
suite: rados:cephadm
suite_branch: pacific-backport-39495-39630-39536-39537-39415-39421-39420-39474-39562-39613-
suite_path: /home/teuthworker/src/github.com_sebastian-philipp_ceph_732027a3aaa7f0273320259c12e605cddca32d95/qa
suite_relpath: qa
suite_repo: http://github.com/sebastian-philipp/ceph.git
suite_sha1: 732027a3aaa7f0273320259c12e605cddca32d95
targets:
  smithi151.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhqoa39vJn0L7XJVNyk3CH0+V2Wa2058gE59I+Z5odSeYdFRL3x2Inx8rzMnbEjJBaeNT0oUf2mNFOmsLhpBXZp/YrJSthV0cIjie8cG/scy3MT2wE6eVsrOHDi/dqMtO0EP3vvNqkZFAgz+mOtKl7xFRgQOHWFT3hks054NvShBNhLKlqMqYPm5eyac0IR3h88UPWOZZhvwfCjfWBYq6vpe5mIsQ/DKb8pt6T/1jlLETZJ8P8mdaiO00U7Qo8opgoCgtA5UhcOzKN+anXzl4lN6ZlckM+q/ykvaSh2lJmX6wjRm/+gG6f+48jHa2RdD/p4cWrWvyh03/2LJ7VcPZ+a9CNTRMfNhvo4+ODszAKc7Rkv5KT1s/RzbEXdw5CALpohq5DcfD05XgQxAOSZ2SgUOXWOVgzfV0UsVHzsxFAgkxjP76agUAKDGyzHMSCoqDdp/zM5GZydyiA0oFDXmGtlCw7JzpkzvPxE1mtjttYj3MmqL6t/A9nPM2Z2X+aTLs=
  smithi169.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnit0wR3qjM0vt8j1Mg3i0fJyFQTyT1SDp1p+B3wtMhO+WDK0V6kdf1t4/Ka7v7QTPeavZcEjlJmJGnxp9lFasm2vIDX5eUGcH4heCtsLntpRgBu9+I6Nn8eQ2TJkeuUbRnzjcucJoYjNul6AhyDxzestaYMevKnm8kW/KDmtfb6lppNimPWbMTKKWjUsUt9Z/VK1HTl3VyGVyDzoCD5kabQFFQGlXjOnoi2T/yepc6FEW5nKU6o3Ghe+TmJJPbfUIUPZBxHbSDqVEcZ3STn+jI+8W60Ri4y53oXAI+0urjyjUhqmPbgZ0tC0HBrI7g7uXnco+lWzsmAKgfTloXXmnfZ98Hfit0TlzBPoMu3ieTudKOaHo2c5pOySzUu6HMVN+lcBLz9GKDbLnPQ/aCw36/ru9Awg7lO+FEhh05ObyndgrfHhFKhp5DmwtvHGrlJJqV+A8kB1E3Ld6P4rteJfoodBKuouTFcP27aWeHVVxn7xfZb/YI7QBSU5ILlOXnLs=
tasks:
- exec:
    all:
    - curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/testing/xUbuntu_20.04/Release.key
      | sudo apt-key add -
    - echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/testing/xUbuntu_20.04/
      /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:testing.list
    - sudo apt update
    - sudo apt -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" 
      install podman
    - echo -e "[[registry]]\nlocation = 'docker.io'\n\n[[registry.mirror]]\nlocation='docker-mirror.front.sepia.ceph.com:5000'\n" 
      | sudo tee /etc/containers/registries.conf
- cephadm:
    conf:
      mgr:
        debug mgr: 20
        debug ms: 1
- cephadm.shell:
    mon.a:
    - ceph orch status
    - ceph orch ps
    - ceph orch ls
    - ceph orch host ls
    - ceph orch device ls
    - ceph orch ls --format yaml
teuthology_branch: master
teuthology_sha1: 3082387bba74fcd24c9700593d10418152d53c97
timestamp: 2021-03-09_14:35:07
tube: smithi
user: swagner
verbose: false
worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.25973

Paddles log¶

2021-03-09 21:50:02,229 DEBUG [paddles.controllers.nodes] Locking 2 smithi nodes for scheduled_swagner@teuthology
2021-03-09 21:50:02,242 INFO  [paddles.controllers.nodes] Locked <Node smithi169.front.sepia.ceph.com> <Node smithi151.front.sepia.ceph.com> for scheduled_swagner@teuthology with description /home/teuthworker/archive/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-
testing-2021-03-09-1019-pacific-distro-basic-smithi/5950060
2021-03-09 21:50:02,293 INFO  [paddles.controllers.jobs] Creating run: swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi
2021-03-09 21:50:02,299 INFO  [paddles.controllers.jobs] Creating job: swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/5950060
[2021-03-09 21:50:02 +0000] [25937] [ERROR] Error handling request
Traceback (most recent call last):
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 130, in handle
    self.handle_request(listener, req, client, addr)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 171, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__
    return self.application(environ, start_response)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/pecan/core.py", line 810, in __call__
    return super(Pecan, self).__call__(environ, start_response)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/pecan/core.py", line 706, in __call__
    self.determine_hooks(state.controller), 'after', state
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/pecan/core.py", line 835, in handle_hooks
    return super(Pecan, self).handle_hooks(hooks, *args, **kw)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/pecan/core.py", line 332, in handle_hooks
    result = getattr(hook, hook_type)(*args)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/pecan/hooks.py", line 218, in after
    self.commit()
  File "/home/ubuntu/paddles/paddles/models/__init__.py", line 108, in commit
    Session.commit()
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 150, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 790, in commit
    self.transaction.commit()
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit
    self._prepare_impl()
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
    self.session.flush()
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush
    self._flush(objects)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush
    flush_context.execute()
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
    rec.execute(self)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
    uow
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
    mapper, table, update)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 692, in _emit_update_statements
    execute(statement, multiparams)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
DataError: (psycopg2.DataError) value too long for type character varying(64)
 [SQL: 'UPDATE jobs SET archive_path=%(archive_path)s, description=%(description)s, email=%(email)s, job_id=%(job_id)s, kernel=%(kernel)s, last_in_suite=%(last_in_suite)s, machine_type=%(machine_type)s, name=%(name)s, nuke_on_error=%(nuke_on_error)s, os_type=%(os_type)s, os_version=%(os_version)s, overrides=%(overrides)s, owner=%(owner)s, roles=%(roles)s, branch=%(branch)s, sha1=%(sha1)s, suite_branch=%(suite_branch)s, suite_sha1=%(suite_sha1)s, targets=%(targets)s, tasks=%(tasks)s, teuthology_branch=%(teuthology_branch)s, "verbose"=%(verbose)s WHERE jobs.id = %(jobs_id)s'] [parameters: {'kernel': u'{"sha1": "distro", "kdb": true}', 'tasks': u'[{"exec": {"all": ["curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/testing/xUbuntu_20.04/Release.key | sudo apt-key add -", "echo \\"deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/testing/xUbuntu_20.04/ /\\" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:testing.list", "sudo apt update", "sudo apt -y -o Dpkg::Options::=\\"--force-confdef\\" -o Dpkg::Options::=\\"--force-confold\\" install podman", "echo -e \\"[[registry]]\\\\nlocation = \'docker.io\'\\\\n\\\\n[[registry.mirror]]\\\\nlocation=\'docker-mirror.front.sepia.ceph.com:5000\'\\\\n\\" | sudo tee /etc/containers/registries.conf"]}}, {"cephadm": {"conf": {"mgr": {"debug ms": 1, "debug mgr": 20}}}}, {"cephadm.shell": {"mon.a": ["ceph orch status", "ceph orch ps", "ceph orch ls", "ceph orch host ls", "ceph orch device ls", "ceph orch ls --format yaml"]}}]', 'verbose': False, 'owner': u'scheduled_swagner@teuthology', 'targets': u'{"ubuntu@smithi169.front.sepia.ceph.com": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5twUzvEhT5jZUwXMB8yBpnctnHrmFl6CIpJsVO2evuyQGjbjhdCnCzVb2svMlp1pniJat7+j988tN1QIevXfanSolHHjF+Jh+wZxKZwKgzUompmqV6u9Ye3NFJtkLzP3qAR1C/2iW/CaEai0ubbg1vMbJC+eQnwK5KR5j1JQYZAgELVI+jq1WKuyrFjkHU5RKwTXBdepH1jY6uvbgC5CUZGfTBrnYEPWxtehc3fi5MvVEYBAtRnVZdW0GtT6l51+bxlJuCsE1OC4EdziuZP67nvO+Oh0Fs70gt6ah0naeecW1hIBlOHrt4RLta5mxlaSTxk5pPVFK+Kz+Od7jk8fRyEAWlnfLbDcbKgg6X9aLYLBDszyYgoxp2SlQBPLDUNCiedgN8SsoeNUFczzx1Xyz+Plrw65Qv4pWrrnAUKWcl1n04ph3Ip3R10Qb0xEUnwCH1YuCsgusmWNGVY1kj3D/9r3/6PvxAB+P/yEJVT2Ielfs7i+yuPEP8NqAqthmLg8=", "ubuntu@smithi151.front.sepia.ceph.com": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCse0FeZ2B7DNqj2oQRy3Tl3NVxUIHVj1O+F69cGGY9gGXw6YcSRMHjzqWIba+/ulS4uLGSZXH+/f9VT/Vkaqwa5iV5VGIowKRthrWKxbYvpKqMga0fADewr4r5jdBO7VgYKLuHhuDo2B81R54EGGGN151ke9WhQ+7xH2lNmW/rnKzi4K0KkxJlJmg1IW2h16R/7qrvNUmvKFTVaJx7/vkcviyh7CUwuuOyB5q3iXI7c4hm97xyyfHpzZeP6ECXPSupEq9I7kbGKfS7gxJ8pweB/rrNqzhYRkf9/J30aQjAK3iugqiVeQYpV6XCExgctgV+UEDILGlOft6q1tnXyke0jYkA5rX7QaGYR4bSiOzRd3D92YrO62MnjRTRJ+P7n2e0umNGCz7vuiVc+ddOzJ99E4cGbgbZ1tE5vVKK8UiHIVFt7L3csCeeyxjArRBCxRnUg4PLgEr3wz+DnQDqyZU2NB/5jncnuxPO69n4Pv4b7VdKBdNlbC8C/BTpWQpoLVM="}', 'job_id': '5950060', 'suite_branch': u'pacific-backport-39495-39630-39536-39537-39415-39421-39420-39474-39562-39613-', 'os_version': u'20.04', 'branch': u'wip-swagner2-testing-2021-03-09-1019-pacific', 'email': u'swagner@suse.com', 'jobs_id': 7269711, 'archive_path': u'/home/teuthworker/archive/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/5950060', 'description': u'rados:cephadm/smoke/{distro/ubuntu_20.04_kubic_testing fixed-2 mon_election/connectivity start}', 'last_in_suite': False, 'machine_type': u'smithi', 'teuthology_branch': u'master', 'sha1': u'b66e292cfec45247a651159e65386f496cb4d718', 'name': u'swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi', 'roles': u'[["mon.a", "mon.c", "mgr.y", "osd.0", "osd.1", "osd.2", "osd.3", "client.0", "ceph.rgw.realm.zone.a", "node-exporter.a", "alertmanager.a"], ["mon.b", "mgr.x", "osd.4", "osd.5", "osd.6", "osd.7", "client.1", "prometheus.a", "grafana.a", "node-exporter.b", "ceph.iscsi.iscsi.a"]]', 'suite_sha1': u'732027a3aaa7f0273320259c12e605cddca32d95', 'nuke_on_error': True, 'os_type': u'ubuntu', 'overrides': u'{"ceph": {"sha1": "b66e292cfec45247a651159e65386f496cb4d718", "log-whitelist": ["\\\\(MDS_ALL_DOWN\\\\)", "\\\\(MDS_UP_LESS_THAN_MAX\\\\)"], "conf": {"mgr": {"debug ms": 1, "debug mgr": 20}, "global": {"mon election default strategy": 3}, "mon": {"debug paxos": 20, "debug mon": 20, "debug ms": 1}, "osd": {"debug osd": 25, "debug filestore": 20, "debug journal": 20, "debug ms": 1, "osd shutdown pgref assert": true}}, "log-ignorelist": ["\\\\(MDS_ALL_DOWN\\\\)", "\\\\(MDS_UP_LESS_THAN_MAX\\\\)"]}, "ceph-deploy": {"conf": {"client": {"log file": "/var/log/ceph/ceph-$name.$pid.log"}, "mon": {"osd default pool size": 2}}}, "workunit": {"sha1": "732027a3aaa7f0273320259c12e605cddca32d95", "branch": "pacific-backport-39495-39630-39536-39537-39415-39421-39420-39474-39562-39613-"}, "install": {"ceph": {"sha1": "b66e292cfec45247a651159e65386f496cb4d718"}}, "admin_socket": {"branch": "wip-swagner2-testing-2021-03-09-1019-pacific"}}'}]

Dispatcher log¶

2021-03-09T21:50:33.095 INFO:teuthology.repo_utils:/home/teuthworker/src/github.com_sebastian-philipp_ceph_732027a3aaa7f0273320259c12e605cddca32d95 was just updated or references a specific commit; assuming it is current
2021-03-09T21:50:33.104 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>

  </body>
</html>

2021-03-09T21:50:33.104 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/report.py", line 498, in try_push_job_info
    push_job_info(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/report.py", line 463, in push_job_info
    reporter.report_job(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/report.py", line 320, in report_job
    response.raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/jobs/
2021-03-09T21:50:33.105 INFO:teuthology.lock.ops:Locking machines...
2021-03-09T21:50:33.143 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>

  </body>
</html>

2021-03-09T21:50:33.143 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/report.py", line 498, in try_push_job_info
    push_job_info(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/report.py", line 463, in push_job_info
    reporter.report_job(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/report.py", line 320, in report_job
    response.raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/jobs/
2021-03-09T21:50:33.170 INFO:teuthology.lock.ops:2 smithi machines locked this try, 2/2 locked so far
2021-03-09T21:50:33.177 WARNING:teuthology.lock.query:Failed to query lock server for status of smithi007.front.sepia.ceph.com
2021-03-09T21:50:33.185 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-dispatcher", line 33, in <module>
    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')())
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/scripts/dispatcher.py", line 33, in main
    teuthology.dispatcher.main(args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/dispatcher/__init__.py", line 132, in main
    job_config = lock_machines(job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/dispatcher/__init__.py", line 179, in lock_machines
    job_config['machine_type'], reimage=False)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/lock/ops.py", line 381, in block_and_lock_machines
    if teuthology.lock.query.is_vm(lmach):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/lock/query.py", line 48, in is_vm
    return status.get('is_vm', False)
AttributeError: 'NoneType' object has no attribute 'get'