ceph-volume output changed: (Expecting value: line 1 column 1 (char 0))
debug 2020-10-06T07:07:33.947+0000 7f428c827700 -1 log_channel(cephadm) log [ERR] : executing refresh((['master'],)) failed. Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line 62, in do_work return f(*arg) File "/usr/share/ceph/mgr/cephadm/module.py", line 1450, in refresh r = self._refresh_host_devices(host) File "/usr/share/ceph/mgr/cephadm/module.py", line 1575, in _refresh_host_devices self.update_osdspec_previews(host) File "/usr/share/ceph/mgr/cephadm/module.py", line 1413, in update_osdspec_previews previews.extend(self.osd_service.get_previews(search_host)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 157, in get_previews return self.generate_previews(osdspecs, host) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 204, in generate_previews concat_out: Dict[str, Any] = json.loads(" ".join(out)) File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) debug 2020-10-06T07:07:33.947+0000 7f4289de2700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'cephadm' while running on mgr.master.rhifho: Expecting value: line 1 column 1 (char 0)
As a result, the deployment of MDSes gets stuck (MDSes never get deployed).
#5 Updated by Nathan Cutler 3 months ago
Joshua Schmid wrote:
Actually this doesn't 'break' the mds deployment. It breaks the mgr, until you restart it.
What's interesting, though, is that it doesn't happen when I run `ceph orch apply -i` on a spec.yaml for deploying MONs, MGRs, and OSDs. It happens after that one completes, all the OSDs are in the tree, and then when I run `ceph orch apply -i` on a spec.yaml that deploys MDSes - that's when it happens.
#7 Updated by Joshua Schmid 3 months ago
Nathan Cutler wrote:
Oh.... maybe it happens at the very end of the OSD deployment. So whichever is the next command, nothing happens because the MGR is out of commission. I just happen to be trying to deploy MDSes at that point.
Something along those lines, yeah..