Bug #48118: JSONDecodeErrors break the server loop - Orchestrator - Ceph

Actions

Copy link

Bug #48118

closed

JSONDecodeErrors break the server loop

Added by Michael Fritch over 3 years ago. Updated over 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Michael Fritch

Category:

cephadm

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

37952

Crash signature (v1):

Crash signature (v2):

Description

a/mgfritch-2020-11-03_14:22:58-rados:cephadm-wip-mgfritch-testing-2020-11-02-1045-distro-basic-smithi/5586920/teuthology.log

2020-11-03T22:12:57.533 INFO:tasks.cephadm:Adding host smithi149 to orchestrator...
2020-11-03T22:12:57.534 INFO:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:9b3ad64c4a7e93bad1c734ad3f5c1c259af6157d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6e34fcca-1e21-11eb-a2ae-001a4aab830c -- ceph orch host add smithi149
2020-11-03T22:12:59.751 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]: debug 2020-11-03T22:12:59.458+0000 7ff559802700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi145'],)) failed.
2020-11-03T22:12:59.752 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]: Traceback (most recent call last):
2020-11-03T22:12:59.752 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/usr/share/ceph/mgr/cephadm/utils.py", line 62, in do_work
2020-11-03T22:12:59.752 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     return f(*arg)
2020-11-03T22:12:59.752 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 132, in refresh
2020-11-03T22:12:59.752 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     r = self._refresh_host_devices(host)
2020-11-03T22:12:59.752 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 260, in _refresh_host_devices
2020-11-03T22:12:59.753 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     devices = json.loads(''.join(out))
2020-11-03T22:12:59.753 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/lib64/python3.6/json/__init__.py", line 354, in loads
2020-11-03T22:12:59.753 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     return _default_decoder.decode(s)
2020-11-03T22:12:59.753 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/lib64/python3.6/json/decoder.py", line 339, in decode
2020-11-03T22:12:59.753 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2020-11-03T22:12:59.753 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/lib64/python3.6/json/decoder.py", line 357, in raw_decode
2020-11-03T22:12:59.754 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     raise JSONDecodeError("Expecting value", s, err.value) from None
2020-11-03T22:12:59.754 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2020-11-03T22:12:59.754 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]: debug 2020-11-03T22:12:59.458+0000 7ff54fdc9700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'cephadm' while running on mgr.y: Expecting value: line 1 column 1 (char 0)
2020-11-03T22:12:59.754 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]: debug 2020-11-03T22:12:59.458+0000 7ff54fdc9700 -1 cephadm.serve:
2020-11-03T22:12:59.754 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]: debug 2020-11-03T22:12:59.458+0000 7ff54fdc9700 -1 Traceback (most recent call last):
2020-11-03T22:12:59.754 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 441, in serve
2020-11-03T22:12:59.754 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     serve.serve()
2020-11-03T22:12:59.755 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 59, in serve
2020-11-03T22:12:59.755 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     self._refresh_hosts_and_daemons()
2020-11-03T22:12:59.755 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 148, in _refresh_hosts_and_daemons
2020-11-03T22:12:59.755 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     refresh(self.mgr.cache.get_hosts())
2020-11-03T22:12:59.755 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/usr/share/ceph/mgr/cephadm/utils.py", line 68, in forall_hosts_wrapper
2020-11-03T22:12:59.755 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     return CephadmOrchestrator.instance._worker_pool.map(do_work, vals)
2020-11-03T22:12:59.755 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map
2020-11-03T22:12:59.756 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     return self._map_async(func, iterable, mapstar, chunksize).get()
2020-11-03T22:12:59.756 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get
2020-11-03T22:12:59.756 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     raise self._value
2020-11-03T22:12:59.756 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker
2020-11-03T22:12:59.756 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     result = (True, func(*args, **kwds))
2020-11-03T22:12:59.756 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar
2020-11-03T22:12:59.757 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     return list(map(*args))
2020-11-03T22:12:59.757 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/usr/share/ceph/mgr/cephadm/utils.py", line 62, in do_work
2020-11-03T22:12:59.757 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     return f(*arg)
2020-11-03T22:12:59.757 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 132, in refresh
2020-11-03T22:12:59.757 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     r = self._refresh_host_devices(host)
2020-11-03T22:12:59.757 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 260, in _refresh_host_devices
2020-11-03T22:12:59.757 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     devices = json.loads(''.join(out))
2020-11-03T22:12:59.758 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/lib64/python3.6/json/__init__.py", line 354, in loads
2020-11-03T22:12:59.758 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     return _default_decoder.decode(s)
2020-11-03T22:12:59.758 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/lib64/python3.6/json/decoder.py", line 339, in decode
2020-11-03T22:12:59.758 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2020-11-03T22:12:59.758 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:   File "/lib64/python3.6/json/decoder.py", line 357, in raw_decode
2020-11-03T22:12:59.758 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]:     raise JSONDecodeError("Expecting value", s, err.value) from None
2020-11-03T22:12:59.758 INFO:journalctl@ceph.mgr.y.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10590]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2020-11-03T22:13:00.250 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]: cephadm 2020-11-03T22:12:59.462011+0000 mgr.y (mgr.14140) 5 : cephadm [ERR] executing refresh((['smithi145'],)) failed.
2020-11-03T22:13:00.251 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]: Traceback (most recent call last):
2020-11-03T22:13:00.251 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]:   File "/usr/share/ceph/mgr/cephadm/utils.py", line 62, in do_work
2020-11-03T22:13:00.251 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]:     return f(*arg)
2020-11-03T22:13:00.252 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 132, in refresh
2020-11-03T22:13:00.252 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]:     r = self._refresh_host_devices(host)
2020-11-03T22:13:00.252 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 260, in _refresh_host_devices
2020-11-03T22:13:00.252 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]:     devices = json.loads(''.join(out))
2020-11-03T22:13:00.252 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]:   File "/lib64/python3.6/json/__init__.py", line 354, in loads
2020-11-03T22:13:00.252 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]:     return _default_decoder.decode(s)
2020-11-03T22:13:00.252 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]:   File "/lib64/python3.6/json/decoder.py", line 339, in decode
2020-11-03T22:13:00.253 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2020-11-03T22:13:00.253 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]:   File "/lib64/python3.6/json/decoder.py", line 357, in raw_decode
2020-11-03T22:13:00.253 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]:     raise JSONDecodeError("Expecting value", s, err.value) from None
2020-11-03T22:13:00.253 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2020-11-03T22:13:00.253 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:12:59 smithi145 bash[10310]: cluster 2020-11-03T22:12:59.462343+0000 mgr.y (mgr.14140) 6 : cluster [ERR] Unhandled exception from module 'cephadm' while running on mgr.y: Expecting value: line 1 column 1 (char 0)
2020-11-03T22:13:02.250 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:13:01 smithi145 bash[10310]: debug 2020-11-03T22:13:01.774+0000 7fd3cd3c8700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) (MGR_MODULE_ERROR)
2020-11-03T22:13:02.251 INFO:journalctl@ceph.mon.a.smithi145.stdout:Nov 03 22:13:01 smithi145 bash[10310]: cluster 2020-11-03T22:13:01.776151+0000 mon.a (mon.0) 126 : cluster [ERR] Health check failed: Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) (MGR_MODULE_ERROR)