Ceph » Orchestrator

Steps to reproduce:¶

• Enable cephadm backend and add a host mgr0
• Create a mgr daemon*

bin/ceph orch apply mgr mgr0

• List services

╰─# bin/ceph orch ls       
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2020-03-06T11:25:09.776+0000 7f31e2004700 -1 WARNING: all dangerous and experimental features are enabled.
2020-03-06T11:25:09.796+0000 7f31e2004700 -1 WARNING: all dangerous and experimental features are enabled.
NAME  RUNNING  REFRESHED  AGE  SPEC     PLACEMENT   IMAGE NAME                                            IMAGE ID      
mgr       1/1  3m age     3m   present  hosts=mgr0  172.17.0.1:5001/ceph/daemon-base:latest-master-devel  33849b8becbf 

• Restart cephadm

# bin/ceph mgr module disable cephadm
# bin/ceph mgr module enable cephadm

• List services again, the SPEC column is gone.

╭─root@ceph-1 /ceph/build ‹master-0306*› 
╰─# bin/ceph orch ls                  
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2020-03-06T11:28:40.058+0000 7facbd562700 -1 WARNING: all dangerous and experimental features are enabled.
2020-03-06T11:28:40.078+0000 7facbd562700 -1 WARNING: all dangerous and experimental features are enabled.
NAME  RUNNING  REFRESHED  AGE  SPEC  PLACEMENT  IMAGE NAME                                            IMAGE ID      
mgr       1/1  22s age    -    -     -          172.17.0.1:5001/ceph/daemon-base:latest-master-devel  33849b8becbf  

• spec dump also returns nothing:

╰─# bin/ceph orch spec dump                                                                                                                                            22 ↵
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2020-03-06T11:29:56.919+0000 7f1ae7279700 -1 WARNING: all dangerous and experimental features are enabled.
2020-03-06T11:29:56.943+0000 7f1ae7279700 -1 WARNING: all dangerous and experimental features are enabled.

Investigation:¶

• Verify spec_store is persisted after module restarted:

╰─# bin/ceph config-key  get mgr/cephadm/spec.mgr
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2020-03-06T11:35:52.866+0000 7fbf07b30700 -1 WARNING: all dangerous and experimental features are enabled.
2020-03-06T11:35:52.890+0000 7fbf07b30700 -1 WARNING: all dangerous and experimental features are enabled.
obtained 'mgr/cephadm/spec.mgr'
{"created": "2020-03-06T11:27:52.870985", "spec": {"placement": {"all_hosts": false, "count": null, "hosts": [["mgr0", "", ""]], "label": null}, "service_id": null, "service_type": "mgr"}}

• mgr.log:

2020-03-06T11:28:39.226+0000 7ff045d4b700  4 mgr get_store_prefix get_store_prefix prefix: mgr/cephadm/host.
2020-03-06T11:28:39.234+0000 7ff045d4b700  0 [cephadm DEBUG root] HostCache.load: host mgr0 has 1 daemons, 1 devices
2020-03-06T11:28:39.234+0000 7ff045d4b700  0 [cephadm DEBUG root] HostCache.load: host osd0 has 0 daemons, 0 devices
2020-03-06T11:28:39.234+0000 7ff045d4b700  4 mgr get_store_prefix get_store_prefix prefix: mgr/cephadm/spec.
2020-03-06T11:28:39.234+0000 7ff045d4b700  0 [cephadm WARNING root] unable to load spec for mgr: expected string or bytes-like object
2020-03-06T11:28:39.234+0000 7ff045d4b700 20 log_channel(cephadm) update_config log_to_monitors {default=true} log_to_syslog {default=false} log_channels {audit=local0,defa
ult=daemon} log_prios {default=info}
2020-03-06T11:28:39.234+0000 7ff045d4b700 10 log_channel(cephadm) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false gray
log_host: 127.0.0.1 graylog_port: 12201)
2020-03-06T11:28:39.234+0000 7ff045d4b700  0 log_channel(cephadm) log [WRN] : unable to load spec for mgr: expected string or bytes-like object

• Raise the exception on https://github.com/ceph/ceph/blob/17e55269bb0fc1f066d74c4eace0956fed0c41a5/src/pybind/mgr/cephadm/module.py#L147-L151 to see the traceback:

2020-03-06T11:33:54.357+0000 7f2bcba79700  0 log_channel(cephadm) log [WRN] : unable to load spec for mgr: expected string or bytes-like object
2020-03-06T11:33:54.357+0000 7f2bcba79700 -1 mgr load Failed to construct class in 'cephadm'
2020-03-06T11:33:54.357+0000 7f2bcba79700 -1 mgr load Traceback (most recent call last):
  File "/ceph/src/pybind/mgr/cephadm/module.py", line 672, in __init__
    self.spec_store.load()
  File "/ceph/src/pybind/mgr/cephadm/module.py", line 151, in load
    raise e
  File "/ceph/src/pybind/mgr/cephadm/module.py", line 142, in load
    spec = ServiceSpec.from_json(v['spec'])
  File "/ceph/src/pybind/mgr/orchestrator/_interface.py", line 1598, in from_json
    v = PlacementSpec.from_dict(v)
  File "/ceph/src/pybind/mgr/orchestrator/_interface.py", line 1281, in from_dict
    _cls = cls(**data)
  File "/ceph/src/pybind/mgr/orchestrator/_interface.py", line 1254, in __init__
    self.hosts = [HostPlacementSpec.parse(x, require_network=False) for x in hosts if x]
  File "/ceph/src/pybind/mgr/orchestrator/_interface.py", line 1254, in <listcomp>
    self.hosts = [HostPlacementSpec.parse(x, require_network=False) for x in hosts if x]
  File "/ceph/src/pybind/mgr/orchestrator/_interface.py", line 77, in parse
    match_host = re.search(host_re, host)
  File "/usr/lib64/python3.7/re.py", line 183, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object