Bug #49710
Updated by David Galloway about 3 years ago
swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/5950060 killed a dispatcher. h3. h1. Job Yaml <pre> archive_path: /home/teuthworker/archive/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/5950060 branch: wip-swagner2-testing-2021-03-09-1019-pacific description: rados:cephadm/smoke/{distro/ubuntu_20.04_kubic_testing fixed-2 mon_election/connectivity start} email: swagner@suse.com first_in_suite: false job_id: '5950060' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-swagner2-testing-2021-03-09-1019-pacific ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 osd shutdown pgref assert: true log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: b66e292cfec45247a651159e65386f496cb4d718 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: b66e292cfec45247a651159e65386f496cb4d718 workunit: branch: pacific-backport-39495-39630-39536-39537-39415-39421-39420-39474-39562-39613- sha1: 732027a3aaa7f0273320259c12e605cddca32d95 owner: scheduled_swagner@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.realm.zone.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a sha1: b66e292cfec45247a651159e65386f496cb4d718 sleep_before_teardown: 0 suite: rados:cephadm suite_branch: pacific-backport-39495-39630-39536-39537-39415-39421-39420-39474-39562-39613- suite_path: /home/teuthworker/src/github.com_sebastian-philipp_ceph_732027a3aaa7f0273320259c12e605cddca32d95/qa suite_relpath: qa suite_repo: http://github.com/sebastian-philipp/ceph.git suite_sha1: 732027a3aaa7f0273320259c12e605cddca32d95 targets: smithi151.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhqoa39vJn0L7XJVNyk3CH0+V2Wa2058gE59I+Z5odSeYdFRL3x2Inx8rzMnbEjJBaeNT0oUf2mNFOmsLhpBXZp/YrJSthV0cIjie8cG/scy3MT2wE6eVsrOHDi/dqMtO0EP3vvNqkZFAgz+mOtKl7xFRgQOHWFT3hks054NvShBNhLKlqMqYPm5eyac0IR3h88UPWOZZhvwfCjfWBYq6vpe5mIsQ/DKb8pt6T/1jlLETZJ8P8mdaiO00U7Qo8opgoCgtA5UhcOzKN+anXzl4lN6ZlckM+q/ykvaSh2lJmX6wjRm/+gG6f+48jHa2RdD/p4cWrWvyh03/2LJ7VcPZ+a9CNTRMfNhvo4+ODszAKc7Rkv5KT1s/RzbEXdw5CALpohq5DcfD05XgQxAOSZ2SgUOXWOVgzfV0UsVHzsxFAgkxjP76agUAKDGyzHMSCoqDdp/zM5GZydyiA0oFDXmGtlCw7JzpkzvPxE1mtjttYj3MmqL6t/A9nPM2Z2X+aTLs= smithi169.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnit0wR3qjM0vt8j1Mg3i0fJyFQTyT1SDp1p+B3wtMhO+WDK0V6kdf1t4/Ka7v7QTPeavZcEjlJmJGnxp9lFasm2vIDX5eUGcH4heCtsLntpRgBu9+I6Nn8eQ2TJkeuUbRnzjcucJoYjNul6AhyDxzestaYMevKnm8kW/KDmtfb6lppNimPWbMTKKWjUsUt9Z/VK1HTl3VyGVyDzoCD5kabQFFQGlXjOnoi2T/yepc6FEW5nKU6o3Ghe+TmJJPbfUIUPZBxHbSDqVEcZ3STn+jI+8W60Ri4y53oXAI+0urjyjUhqmPbgZ0tC0HBrI7g7uXnco+lWzsmAKgfTloXXmnfZ98Hfit0TlzBPoMu3ieTudKOaHo2c5pOySzUu6HMVN+lcBLz9GKDbLnPQ/aCw36/ru9Awg7lO+FEhh05ObyndgrfHhFKhp5DmwtvHGrlJJqV+A8kB1E3Ld6P4rteJfoodBKuouTFcP27aWeHVVxn7xfZb/YI7QBSU5ILlOXnLs= tasks: - exec: all: - curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/testing/xUbuntu_20.04/Release.key | sudo apt-key add - - echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/testing/xUbuntu_20.04/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:testing.list - sudo apt update - sudo apt -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install podman - echo -e "[[registry]]\nlocation = 'docker.io'\n\n[[registry.mirror]]\nlocation='docker-mirror.front.sepia.ceph.com:5000'\n" | sudo tee /etc/containers/registries.conf - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml teuthology_branch: master teuthology_sha1: 3082387bba74fcd24c9700593d10418152d53c97 timestamp: 2021-03-09_14:35:07 tube: smithi user: swagner verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.25973 </pre> h3. h1. Paddles log <pre> 2021-03-09 21:50:02,229 DEBUG [paddles.controllers.nodes] Locking 2 smithi nodes for scheduled_swagner@teuthology 2021-03-09 21:50:02,242 INFO [paddles.controllers.nodes] Locked <Node smithi169.front.sepia.ceph.com> <Node smithi151.front.sepia.ceph.com> for scheduled_swagner@teuthology with description /home/teuthworker/archive/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2- testing-2021-03-09-1019-pacific-distro-basic-smithi/5950060 2021-03-09 21:50:02,293 INFO [paddles.controllers.jobs] Creating run: swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi 2021-03-09 21:50:02,299 INFO [paddles.controllers.jobs] Creating job: swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/5950060 [2021-03-09 21:50:02 +0000] [25937] [ERROR] Error handling request Traceback (most recent call last): File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 130, in handle self.handle_request(listener, req, client, addr) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 171, in handle_request respiter = self.wsgi(environ, resp.start_response) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__ return self.application(environ, start_response) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/pecan/core.py", line 810, in __call__ return super(Pecan, self).__call__(environ, start_response) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/pecan/core.py", line 706, in __call__ self.determine_hooks(state.controller), 'after', state File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/pecan/core.py", line 835, in handle_hooks return super(Pecan, self).handle_hooks(hooks, *args, **kw) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/pecan/core.py", line 332, in handle_hooks result = getattr(hook, hook_type)(*args) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/pecan/hooks.py", line 218, in after self.commit() File "/home/ubuntu/paddles/paddles/models/__init__.py", line 108, in commit Session.commit() File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 150, in do return getattr(self.registry(), name)(*args, **kwargs) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 790, in commit self.transaction.commit() File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit self._prepare_impl() File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl self.session.flush() File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush self._flush(objects) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush transaction.rollback(_capture_exception=True) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush flush_context.execute() File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute rec.execute(self) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute uow File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj mapper, table, update) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 692, in _emit_update_statements execute(statement, multiparams) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute return meth(self, multiparams, params) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement compiled_sql, distilled_params File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context context) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception exc_info File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context context) File "/home/ubuntu/.virtualenvs/paddles/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute cursor.execute(statement, parameters) DataError: (psycopg2.DataError) value too long for type character varying(64) [SQL: 'UPDATE jobs SET archive_path=%(archive_path)s, description=%(description)s, email=%(email)s, job_id=%(job_id)s, kernel=%(kernel)s, last_in_suite=%(last_in_suite)s, machine_type=%(machine_type)s, name=%(name)s, nuke_on_error=%(nuke_on_error)s, os_type=%(os_type)s, os_version=%(os_version)s, overrides=%(overrides)s, owner=%(owner)s, roles=%(roles)s, branch=%(branch)s, sha1=%(sha1)s, suite_branch=%(suite_branch)s, suite_sha1=%(suite_sha1)s, targets=%(targets)s, tasks=%(tasks)s, teuthology_branch=%(teuthology_branch)s, "verbose"=%(verbose)s WHERE jobs.id = %(jobs_id)s'] [parameters: {'kernel': u'{"sha1": "distro", "kdb": true}', 'tasks': u'[{"exec": {"all": ["curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/testing/xUbuntu_20.04/Release.key | sudo apt-key add -", "echo \\"deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/testing/xUbuntu_20.04/ /\\" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:testing.list", "sudo apt update", "sudo apt -y -o Dpkg::Options::=\\"--force-confdef\\" -o Dpkg::Options::=\\"--force-confold\\" install podman", "echo -e \\"[[registry]]\\\\nlocation = \'docker.io\'\\\\n\\\\n[[registry.mirror]]\\\\nlocation=\'docker-mirror.front.sepia.ceph.com:5000\'\\\\n\\" | sudo tee /etc/containers/registries.conf"]}}, {"cephadm": {"conf": {"mgr": {"debug ms": 1, "debug mgr": 20}}}}, {"cephadm.shell": {"mon.a": ["ceph orch status", "ceph orch ps", "ceph orch ls", "ceph orch host ls", "ceph orch device ls", "ceph orch ls --format yaml"]}}]', 'verbose': False, 'owner': u'scheduled_swagner@teuthology', 'targets': u'{"ubuntu@smithi169.front.sepia.ceph.com": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5twUzvEhT5jZUwXMB8yBpnctnHrmFl6CIpJsVO2evuyQGjbjhdCnCzVb2svMlp1pniJat7+j988tN1QIevXfanSolHHjF+Jh+wZxKZwKgzUompmqV6u9Ye3NFJtkLzP3qAR1C/2iW/CaEai0ubbg1vMbJC+eQnwK5KR5j1JQYZAgELVI+jq1WKuyrFjkHU5RKwTXBdepH1jY6uvbgC5CUZGfTBrnYEPWxtehc3fi5MvVEYBAtRnVZdW0GtT6l51+bxlJuCsE1OC4EdziuZP67nvO+Oh0Fs70gt6ah0naeecW1hIBlOHrt4RLta5mxlaSTxk5pPVFK+Kz+Od7jk8fRyEAWlnfLbDcbKgg6X9aLYLBDszyYgoxp2SlQBPLDUNCiedgN8SsoeNUFczzx1Xyz+Plrw65Qv4pWrrnAUKWcl1n04ph3Ip3R10Qb0xEUnwCH1YuCsgusmWNGVY1kj3D/9r3/6PvxAB+P/yEJVT2Ielfs7i+yuPEP8NqAqthmLg8=", "ubuntu@smithi151.front.sepia.ceph.com": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCse0FeZ2B7DNqj2oQRy3Tl3NVxUIHVj1O+F69cGGY9gGXw6YcSRMHjzqWIba+/ulS4uLGSZXH+/f9VT/Vkaqwa5iV5VGIowKRthrWKxbYvpKqMga0fADewr4r5jdBO7VgYKLuHhuDo2B81R54EGGGN151ke9WhQ+7xH2lNmW/rnKzi4K0KkxJlJmg1IW2h16R/7qrvNUmvKFTVaJx7/vkcviyh7CUwuuOyB5q3iXI7c4hm97xyyfHpzZeP6ECXPSupEq9I7kbGKfS7gxJ8pweB/rrNqzhYRkf9/J30aQjAK3iugqiVeQYpV6XCExgctgV+UEDILGlOft6q1tnXyke0jYkA5rX7QaGYR4bSiOzRd3D92YrO62MnjRTRJ+P7n2e0umNGCz7vuiVc+ddOzJ99E4cGbgbZ1tE5vVKK8UiHIVFt7L3csCeeyxjArRBCxRnUg4PLgEr3wz+DnQDqyZU2NB/5jncnuxPO69n4Pv4b7VdKBdNlbC8C/BTpWQpoLVM="}', 'job_id': '5950060', 'suite_branch': u'pacific-backport-39495-39630-39536-39537-39415-39421-39420-39474-39562-39613-', 'os_version': u'20.04', 'branch': u'wip-swagner2-testing-2021-03-09-1019-pacific', 'email': u'swagner@suse.com', 'jobs_id': 7269711, 'archive_path': u'/home/teuthworker/archive/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/5950060', 'description': u'rados:cephadm/smoke/{distro/ubuntu_20.04_kubic_testing fixed-2 mon_election/connectivity start}', 'last_in_suite': False, 'machine_type': u'smithi', 'teuthology_branch': u'master', 'sha1': u'b66e292cfec45247a651159e65386f496cb4d718', 'name': u'swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi', 'roles': u'[["mon.a", "mon.c", "mgr.y", "osd.0", "osd.1", "osd.2", "osd.3", "client.0", "ceph.rgw.realm.zone.a", "node-exporter.a", "alertmanager.a"], ["mon.b", "mgr.x", "osd.4", "osd.5", "osd.6", "osd.7", "client.1", "prometheus.a", "grafana.a", "node-exporter.b", "ceph.iscsi.iscsi.a"]]', 'suite_sha1': u'732027a3aaa7f0273320259c12e605cddca32d95', 'nuke_on_error': True, 'os_type': u'ubuntu', 'overrides': u'{"ceph": {"sha1": "b66e292cfec45247a651159e65386f496cb4d718", "log-whitelist": ["\\\\(MDS_ALL_DOWN\\\\)", "\\\\(MDS_UP_LESS_THAN_MAX\\\\)"], "conf": {"mgr": {"debug ms": 1, "debug mgr": 20}, "global": {"mon election default strategy": 3}, "mon": {"debug paxos": 20, "debug mon": 20, "debug ms": 1}, "osd": {"debug osd": 25, "debug filestore": 20, "debug journal": 20, "debug ms": 1, "osd shutdown pgref assert": true}}, "log-ignorelist": ["\\\\(MDS_ALL_DOWN\\\\)", "\\\\(MDS_UP_LESS_THAN_MAX\\\\)"]}, "ceph-deploy": {"conf": {"client": {"log file": "/var/log/ceph/ceph-$name.$pid.log"}, "mon": {"osd default pool size": 2}}}, "workunit": {"sha1": "732027a3aaa7f0273320259c12e605cddca32d95", "branch": "pacific-backport-39495-39630-39536-39537-39415-39421-39420-39474-39562-39613-"}, "install": {"ceph": {"sha1": "b66e292cfec45247a651159e65386f496cb4d718"}}, "admin_socket": {"branch": "wip-swagner2-testing-2021-03-09-1019-pacific"}}'}] </pre> h3. h1. Dispatcher log <pre> 2021-03-09T21:50:33.095 INFO:teuthology.repo_utils:/home/teuthworker/src/github.com_sebastian-philipp_ceph_732027a3aaa7f0273320259c12e605cddca32d95 was just updated or references a specific commit; assuming it is current 2021-03-09T21:50:33.104 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/jobs/ failed with status 500: <html> <head> <title>Internal Server Error</title> </head> <body> <h1><p>Internal Server Error</p></h1> </body> </html> 2021-03-09T21:50:33.104 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/ Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/report.py", line 498, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/report.py", line 463, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/report.py", line 320, in report_job response.raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/jobs/ 2021-03-09T21:50:33.105 INFO:teuthology.lock.ops:Locking machines... 2021-03-09T21:50:33.143 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/jobs/ failed with status 500: <html> <head> <title>Internal Server Error</title> </head> <body> <h1><p>Internal Server Error</p></h1> </body> </html> 2021-03-09T21:50:33.143 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/ Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/report.py", line 498, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/report.py", line 463, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/report.py", line 320, in report_job response.raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/swagner-2021-03-09_14:35:07-rados:cephadm-wip-swagner2-testing-2021-03-09-1019-pacific-distro-basic-smithi/jobs/ 2021-03-09T21:50:33.170 INFO:teuthology.lock.ops:2 smithi machines locked this try, 2/2 locked so far 2021-03-09T21:50:33.177 WARNING:teuthology.lock.query:Failed to query lock server for status of smithi007.front.sepia.ceph.com 2021-03-09T21:50:33.185 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-dispatcher", line 33, in <module> sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/dispatcher/__init__.py", line 132, in main job_config = lock_machines(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/dispatcher/__init__.py", line 179, in lock_machines job_config['machine_type'], reimage=False) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/lock/ops.py", line 381, in block_and_lock_machines if teuthology.lock.query.is_vm(lmach): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/lock/query.py", line 48, in is_vm return status.get('is_vm', False) AttributeError: 'NoneType' object has no attribute 'get' </pre>