Project

General

Profile

Actions
Copy link

Bug #48120

closed

JSONDecodeErrors unclear during cephadm upgrade

Added by Michael Fritch over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Michael Fritch
Category:
cephadm
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
37952
Crash signature (v1):
Crash signature (v2):

Description

a/teuthology/mgfritch-2020-11-03_14:22:58-rados:cephadm-wip-mgfritch-testing-2020-11-02-1045-distro-basic-smithi/5586898/teuthology.log

During a cephadm upgrade it's unclear that the cause of the JSONDecodeError is the 'Pulling container...' log line that proceeds the JSON output:

2020-11-03T21:50:25.969 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]: debug 2020-11-03T21:50:25.644+0000 7f1e313d2700 -1 log_channel(cephadm) log [ERR] : Failed to pull quay.ceph.io/ceph-ci/ceph:9b3ad64c4a7e93bad1c734ad3f5c1c259af6157d on smithi069: Pulling container image quay.ceph.io/ceph-ci/ceph:9b3ad64c4a7e93bad1c734ad3f5c1c259af6157d...
2020-11-03T21:50:25.970 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]: {
2020-11-03T21:50:25.970 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]:     "ceph_version": "ceph version 16.0.0-6825-g9b3ad64c (9b3ad64c4a7e93bad1c734ad3f5c1c259af6157d) pacific (dev)",
2020-11-03T21:50:25.970 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]:     "image_id": "d2aeb735616950e453a71129ae7a9038ea47d82ddf0657e1f91799caa94aeffd",
2020-11-03T21:50:25.970 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]:     "repo_digest": "quay.ceph.io/ceph-ci/ceph@sha256:fd8aaf39d3ef0220f4af30440c34594e573ef61aecfcde6ffcd25af803c51777" 
2020-11-03T21:50:25.971 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]: }
2020-11-03T21:50:25.971 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]: Traceback (most recent call last):
2020-11-03T21:50:25.971 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 2116, in _get_container_image_info
2020-11-03T21:50:25.971 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]:     j = json.loads('\n'.join(out))
2020-11-03T21:50:25.972 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]:   File "/lib64/python3.6/json/__init__.py", line 354, in loads
2020-11-03T21:50:25.972 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]:     return _default_decoder.decode(s)
2020-11-03T21:50:25.972 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]:   File "/lib64/python3.6/json/decoder.py", line 339, in decode
2020-11-03T21:50:25.972 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2020-11-03T21:50:25.973 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]:   File "/lib64/python3.6/json/decoder.py", line 357, in raw_decode
2020-11-03T21:50:25.973 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]:     raise JSONDecodeError("Expecting value", s, err.value) from None
2020-11-03T21:50:25.973 INFO:journalctl@ceph.mgr.x.smithi075.stdout:Nov 03 21:50:25 smithi075 bash[19699]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2020-11-03T21:50:26.469 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]: cephadm 2020-11-03T21:50:25.648591+0000 mgr.x (mgr.34502) 4 : cephadm [ERR] Failed to pull quay.ceph.io/ceph-ci/ceph:9b3ad64c4a7e93bad1c734ad3f5c1c259af6157d on smithi069: Pulling container image quay.ceph.io/ceph-ci/ceph:9b3ad64c4a7e93bad1c734ad3f5c1c259af6157d...
2020-11-03T21:50:26.469 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]: {
2020-11-03T21:50:26.470 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]:     "ceph_version": "ceph version 16.0.0-6825-g9b3ad64c (9b3ad64c4a7e93bad1c734ad3f5c1c259af6157d) pacific (dev)",
2020-11-03T21:50:26.470 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]:     "image_id": "d2aeb735616950e453a71129ae7a9038ea47d82ddf0657e1f91799caa94aeffd",
2020-11-03T21:50:26.470 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]:     "repo_digest": "quay.ceph.io/ceph-ci/ceph@sha256:fd8aaf39d3ef0220f4af30440c34594e573ef61aecfcde6ffcd25af803c51777" 
2020-11-03T21:50:26.470 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]: }
2020-11-03T21:50:26.470 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]: Traceback (most recent call last):
2020-11-03T21:50:26.470 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 2116, in _get_container_image_info
2020-11-03T21:50:26.470 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]:     j = json.loads('\n'.join(out))
2020-11-03T21:50:26.471 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]:   File "/lib64/python3.6/json/__init__.py", line 354, in loads
2020-11-03T21:50:26.471 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]:     return _default_decoder.decode(s)
2020-11-03T21:50:26.471 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]:   File "/lib64/python3.6/json/decoder.py", line 339, in decode
2020-11-03T21:50:26.471 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2020-11-03T21:50:26.471 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]:   File "/lib64/python3.6/json/decoder.py", line 357, in raw_decode
2020-11-03T21:50:26.471 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]:     raise JSONDecodeError("Expecting value", s, err.value) from None
2020-11-03T21:50:26.471 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2020-11-03T21:50:26.472 INFO:journalctl@ceph.mon.b.smithi075.stdout:Nov 03 21:50:26 smithi075 bash[8095]: audit 2020-11-03T21:50:25.651265+0000 mon.b (mon.2) 65 : audit [DBG] from='mgr.34502 172.21.15.75:0/1073205177' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch
Actions
Copy link
 #1

Updated by Michael Fritch over 3 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 37952
Actions
Copy link
 #2

Updated by Sebastian Wagner over 3 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF