Ceph » Orchestrator

I have restarted the mgr server itself just to make sure the docker did fully reload, and tried to re-run the upgrade command and have uploaded and updated log file for the whole of today.

I did notice lots of "mgr finish mon failed to return metadata for mds.cephfs.sn-s01.snkhfd", and have noticed that no stats are showing in Ceph Dashboard - > Filesystems where the normal performance graphs and request data would show for cephfs.
Not sure if related or due to a part complete upgrade currently.

Sep 22 15:20:25 sn-m01 bash[1694]: debug 2020-09-22T15:20:25.586+0000 7f710dc7f700 -1 mgr load Traceback (most recent call last):
Sep 22 15:20:25 sn-m01 bash[1694]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 312, in __init__
Sep 22 15:20:25 sn-m01 bash[1694]:     self.upgrade = CephadmUpgrade(self)
Sep 22 15:20:25 sn-m01 bash[1694]:   File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 59, in __init__
Sep 22 15:20:25 sn-m01 bash[1694]:     self.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t))
Sep 22 15:20:25 sn-m01 bash[1694]:   File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 50, in from_json
Sep 22 15:20:25 sn-m01 bash[1694]:     return cls(**data)
Sep 22 15:20:25 sn-m01 bash[1694]: TypeError: type object argument after ** must be a mapping, not NoneType

Attached.

Thanks

Having same problem here:
added new host & osds yesterday evening. while the cluster was still rebalancing removed all OSDs from another Host via "for i in {51..58}; do ceph orch osd rm $i; done"
today wanted to check the status with "ceph orch osd rm status" but got "Error ENOENT: Module not found"

mgr restart did not help. checking the logs found some stuff like

Oct 06 08:56:37 one1-ceph1 bash⁷¹⁵²⁶: debug 2020-10-06T06:56:37.368+0000 7f6049a68700 -1 mgr load Failed to construct class in 'cephadm'
Oct 06 08:56:37 one1-ceph1 bash⁷¹⁵²⁶: debug 2020-10-06T06:56:37.368+0000 7f6049a68700 -1 mgr load Traceback (most recent call last):
Oct 06 08:56:37 one1-ceph1 bash⁷¹⁵²⁶: File "/usr/share/ceph/mgr/cephadm/module.py", line 325, in init
Oct 06 08:56:37 one1-ceph1 bash⁷¹⁵²⁶: self.rm_util.load_from_store()
Oct 06 08:56:37 one1-ceph1 bash⁷¹⁵²⁶: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 465, in load_from_store
Oct 06 08:56:37 one1-ceph1 bash⁷¹⁵²⁶: osd_obj = OSD.from_json(json.loads(osd), ctx=self)
Oct 06 08:56:37 one1-ceph1 bash⁷¹⁵²⁶: File "/lib64/python3.6/json/__init__.py", line 348, in loads
Oct 06 08:56:37 one1-ceph1 bash⁷¹⁵²⁶: 'not {!r}'.format(s.__class__.__name__))
Oct 06 08:56:37 one1-ceph1 bash⁷¹⁵²⁶: TypeError: the JSON object must be str, bytes or bytearray, not 'dict'
Oct 06 08:56:37 one1-ceph1 bash⁷¹⁵²⁶: debug 2020-10-06T06:56:37.368+0000 7f6049a68700 -1 mgr operator() Failed to run module in active mode ('cephadm')
...
Oct 06 08:56:38 one1-ceph1 bash⁷¹⁵²⁶: debug 2020-10-06T06:56:38.097+0000 7f6049a68700 -1 no module 'cephadm'
Oct 06 08:56:38 one1-ceph1 bash⁷¹⁵²⁶: debug 2020-10-06T06:56:38.097+0000 7f6049a68700 -1 mgr.server reply reply (2) No such file or directory Module not found
...
Oct 06 09:04:10 one1-ceph1 bash⁷¹⁵²⁶: debug 2020-10-06T07:04:10.708+0000 7f604b26b700 0 mgr set_store `config-key set mgr/insights/health_history/2020-10-06_07 {"checks": {"BLUEFS_SPILLOVER": {"HEALTH_WARN": {"summary": ["1 OSD experiencing BlueFS spillover"], "detail": [" osd.55 spilled over 166 MiB metada>
Oct 06 09:04:10 one1-ceph1 bash⁷¹⁵²⁶: undersized for 65s, current state active+undersized+degraded, last acting [9,19]", "pg 20.1fb is active+undersized+degraded, acting [14,19]", "pg 20.1d0 is stuck undersized for 81s, current state active+undersized+degraded, last acting [23,10]", "pg 20.1da is stuck undersize>
Oct 06 09:04:10 one1-ceph1 bash⁷¹⁵²⁶: debug 2020-10-06T07:04:10.708+0000 7f604b26b700 0 mgr set_store mon returned -27: error: entry size limited to 65536 bytes. Use 'mon config key max entry size' to manually adjust

checked config key for OSD removal (ceph config-key get mgr/cephadm/osd_remove_queue)

obtained 'mgr/cephadm/osd_remove_queue'
[{"osd_id": 51, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "nodename": "one1-ceph3", "drain_started_at": "2020-10-05T16:56:41.888866", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2020-10-05T16:56:35.315613"}, {"osd_id": 52, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "nodename": "one1-ceph3", "drain_started_at": "2020-10-05T16:56:42.949972", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2020-10-05T16:56:36.090281"}, {"osd_id": 53, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "nodename": "one1-ceph3", "drain_started_at": "2020-10-05T16:56:43.976747", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2020-10-05T16:56:36.869471"}, {"osd_id": 54, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "nodename": "one1-ceph3", "drain_started_at": "2020-10-05T16:56:44.730852", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2020-10-05T16:56:37.724410"}, {"osd_id": 55, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "nodename": "one1-ceph3", "drain_started_at": "2020-10-05T16:56:45.764194", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2020-10-05T16:56:38.471945"}, {"osd_id": 56, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "nodename": "one1-ceph3", "drain_started_at": "2020-10-05T16:56:46.793051", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2020-10-05T16:56:39.214927"}, {"osd_id": 57, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "nodename": "one1-ceph3", "drain_started_at": "2020-10-05T16:56:47.813943", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2020-10-05T16:56:39.957469"}, {"osd_id": 58, "started": true, "draining": true, "stopped": false, "replace": false, "force": false, "nodename": "one1-ceph3", "drain_started_at": "2020-10-05T16:56:48.845405", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2020-10-05T16:56:40.703363"}]

Is there any workaround as this is a production cluster? thanks.