Bug #47760: ceph-volume output changed: (Expecting value: line 1 column 1 (char 0)) - ceph-volume - Ceph

Actions

Copy link

Bug #47760

closed

ceph-volume output changed: (Expecting value: line 1 column 1 (char 0))

Added by Joshua Schmid over 3 years ago. Updated over 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Jan Fajerski

Target version:

% Done:

Source:

Development

Tags:

Backport:

octopus,nautilus

Regression:

Severity:

1 - critical

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

37591

Crash signature (v1):

Crash signature (v2):

Description

debug 2020-10-06T07:07:33.947+0000 7f428c827700 -1 log_channel(cephadm) log [ERR] : executing refresh((['master'],)) failed.
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/utils.py", line 62, in do_work
    return f(*arg)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1450, in refresh
    r = self._refresh_host_devices(host)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1575, in _refresh_host_devices
    self.update_osdspec_previews(host)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1413, in update_osdspec_previews
    previews.extend(self.osd_service.get_previews(search_host))
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 157, in get_previews
    return self.generate_previews(osdspecs, host)
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 204, in generate_previews
    concat_out: Dict[str, Any] = json.loads(" ".join(out))
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
debug 2020-10-06T07:07:33.947+0000 7f4289de2700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'cephadm' while running on mgr.master.rhifho: Expecting value: line 1 column 1 (char 0)

As a result, the deployment of MDSes gets stuck (MDSes never get deployed).

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Nathan Cutler over 3 years ago

Description updated (diff)

Actions

Copy link

Updated by Nathan Cutler over 3 years ago

Description updated (diff)
Severity changed from 3 - minor to 1 - critical

Raising severity since this breaks the MDS deployment.

Actions

Copy link

Updated by Joshua Schmid over 3 years ago

Actually this doesn't 'break' the mds deployment. It breaks the mgr, until you restart it.

Actions

Copy link

Updated by Jan Fajerski over 3 years ago

Related to Bug #47729: ceph-volume lvm batch --report json returns invalid json when empty added

Actions

Copy link

Updated by Nathan Cutler over 3 years ago

Joshua Schmid wrote:

Actually this doesn't 'break' the mds deployment. It breaks the mgr, until you restart it.

What's interesting, though, is that it doesn't happen when I run `ceph orch apply -i` on a spec.yaml for deploying MONs, MGRs, and OSDs. It happens after that one completes, all the OSDs are in the tree, and then when I run `ceph orch apply -i` on a spec.yaml that deploys MDSes - that's when it happens.

Actions

Copy link

Updated by Nathan Cutler over 3 years ago

Oh.... maybe it happens at the very end of the OSD deployment. So whichever is the next command, nothing happens because the MGR is out of commission. I just happen to be trying to deploy MDSes at that point.

Actions

Copy link

Updated by Joshua Schmid over 3 years ago

Nathan Cutler wrote:

Oh.... maybe it happens at the very end of the OSD deployment. So whichever is the next command, nothing happens because the MGR is out of commission. I just happen to be trying to deploy MDSes at that point.

Something along those lines, yeah..

Actions

Copy link