Project

General

Profile

Bug #47760

ceph-volume output changed: (Expecting value: line 1 column 1 (char 0))

Added by Joshua Schmid 3 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
octopus,nautilus
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

debug 2020-10-06T07:07:33.947+0000 7f428c827700 -1 log_channel(cephadm) log [ERR] : executing refresh((['master'],)) failed.
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/utils.py", line 62, in do_work
    return f(*arg)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1450, in refresh
    r = self._refresh_host_devices(host)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1575, in _refresh_host_devices
    self.update_osdspec_previews(host)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1413, in update_osdspec_previews
    previews.extend(self.osd_service.get_previews(search_host))
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 157, in get_previews
    return self.generate_previews(osdspecs, host)
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 204, in generate_previews
    concat_out: Dict[str, Any] = json.loads(" ".join(out))
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
debug 2020-10-06T07:07:33.947+0000 7f4289de2700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'cephadm' while running on mgr.master.rhifho: Expecting value: line 1 column 1 (char 0)

As a result, the deployment of MDSes gets stuck (MDSes never get deployed).


Related issues

Related to ceph-volume - Bug #47729: ceph-volume lvm batch --report json returns invalid json when empty Resolved

History

#1 Updated by Nathan Cutler 3 months ago

  • Description updated (diff)

#2 Updated by Nathan Cutler 3 months ago

  • Description updated (diff)
  • Severity changed from 3 - minor to 1 - critical

Raising severity since this breaks the MDS deployment.

#3 Updated by Joshua Schmid 3 months ago

Actually this doesn't 'break' the mds deployment. It breaks the mgr, until you restart it.

#4 Updated by Jan Fajerski 3 months ago

  • Related to Bug #47729: ceph-volume lvm batch --report json returns invalid json when empty added

#5 Updated by Nathan Cutler 3 months ago

Joshua Schmid wrote:

Actually this doesn't 'break' the mds deployment. It breaks the mgr, until you restart it.

What's interesting, though, is that it doesn't happen when I run `ceph orch apply -i` on a spec.yaml for deploying MONs, MGRs, and OSDs. It happens after that one completes, all the OSDs are in the tree, and then when I run `ceph orch apply -i` on a spec.yaml that deploys MDSes - that's when it happens.

#6 Updated by Nathan Cutler 3 months ago

Oh.... maybe it happens at the very end of the OSD deployment. So whichever is the next command, nothing happens because the MGR is out of commission. I just happen to be trying to deploy MDSes at that point.

#7 Updated by Joshua Schmid 3 months ago

Nathan Cutler wrote:

Oh.... maybe it happens at the very end of the OSD deployment. So whichever is the next command, nothing happens because the MGR is out of commission. I just happen to be trying to deploy MDSes at that point.

Something along those lines, yeah..

#8 Updated by Jan Fajerski 3 months ago

  • Project changed from Orchestrator to ceph-volume
  • Category deleted (cephadm/osd)
  • Status changed from New to Fix Under Review
  • Assignee set to Jan Fajerski
  • Backport set to octopus,nautilus
  • Pull request ID set to 37591

#9 Updated by Nathan Cutler 3 months ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF