Bug #62297
opennfs cluster commands returns Error EINVAL: 'exporter'
0%
Description
TL;DR: nfs commands like `ceph nfs cluster ls` fail in rook orchestrated cluster; backtrace is full of orch and rook code references in mgr log¶
I orchestrated a cluster using rook (I use yamls from [0]), I tried out a pretty trivial command `ceph nfs cluster ls` for which i got an error:
sh-4.4$ ceph -v
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
sh-4.4$ ceph nfs cluster ls
Error EINVAL: 'exporter'
I use these YAMLs to create a cluster:
crds.yaml, common.yaml, operator.yaml, toolbox.yaml, cluster-test.yaml, filesystem-test.yaml, nfs-test.yaml
When i grep for quay images used in these YAMLs:
cluster-test.yaml:32: image: quay.io/ceph/ceph:v18
toolbox.yaml:21: image: quay.io/ceph/ceph:v17.2.6
there is difference in the versions so i thought maybe because it's this discrepancy that might've lead to an issue, but I wanted to confirm it so went ahead and changed the cluster-test.yaml to use v17.2.6 so now both the YAMLs report same version:
cluster-test.yaml:32: image: quay.io/ceph/ceph:v17.2.6
toolbox.yaml:21: image: quay.io/ceph/ceph:v17.2.6
and this time it works, bingo
sh-4.4$ ceph -v
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
sh-4.4$ ceph nfs cluster ls
my-nfs
but i was curious if there is regression with reef release or not so changed the YAMLs to report v18.0:
cluster-test.yaml:32: image: quay.io/ceph/ceph:v18
toolbox.yaml:21: image: quay.io/ceph/ceph:v18
and my suspicion was right
sh-4.4$ ceph -v
ceph version 18.1.3 (f594a0802c34733bb06e5993bc4bdb085c9a5f3f) reef (rc)
sh-4.4$ ceph nfs cluster ls
Error EINVAL: 'exporter'
fails for cluster creation cmd too
sh-4.4$ ceph nfs cluster create cephfs-nfs
Error EINVAL: 'exporter'
So it seems like there is regression in reef, I dug deep and it looks like daemon_type_to_service() in pybind/mgr/orchestrator/_interface.py is provided with a key not present in it's map.
The code flow is _cmd_nfs_cluster_ls() -> available_clusters() -> describe_service() line 420 [1] -> service_name() -> daemon_type_to_service()
I suspect issue to be somewhere around [1].
PS: Attached mgr log with the tracker.
[0] https://github.com/rook/rook/tree/master/deploy/examples
[1] https://github.com/ceph/ceph/blob/main/src/pybind/mgr/rook/module.py#L420
Files
Updated by Venky Shankar 9 months ago
Dhairya shared the backtrace in cephfs slack channel
debug 2023-08-02T13:38:34.694+0000 7f8f7d071700 0 [rook INFO rook.rook_cluster] Full fetch of <bound method CustomObjectsApi.list_namespaced_custom_object of <kubernetes.client.api.custom_objects_api.CustomObjectsApi object at 0x7f8f72019fd0>>. result: 1
debug 2023-08-02T13:38:34.709+0000 7f8f7d071700 0 [rook INFO rook.rook_cluster] Full fetch of <bound method CoreV1Api.list_namespaced_pod of <kubernetes.client.api.core_v1_api.CoreV1Api object at 0x7f8f93bd8ef0>>. result: 8
debug 2023-08-02T13:38:34.716+0000 7f8f7d071700 0 [rook ERROR orchestrator._interface] 'exporter'
Traceback (most recent call last):
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper
return OrchResult(f(*args, **kwargs))
File "/usr/share/ceph/mgr/rook/module.py", line 420, in describe_service
if dd.service_name() not in spec:
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1111, in service_name
if daemon_type_to_service(self.daemon_type) in ServiceSpec.REQUIRES_SERVICE_ID:
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 811, in daemon_type_to_service
return mapping[dtype]
KeyError: 'exporter'
debug 2023-08-02T13:38:34.716+0000 7f8f7d071700 0 [nfs ERROR nfs.cluster] Failed to list NFS Cluster
Traceback (most recent call last):
File "/usr/share/ceph/mgr/nfs/cluster.py", line 173, in list_nfs_cluster
return available_clusters(self.mgr)
File "/usr/share/ceph/mgr/nfs/utils.py", line 71, in available_clusters
orchestrator.raise_if_exception(completion)
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 228, in raise_if_exception
raise e
KeyError: 'exporter'
debug 2023-08-02T13:38:34.716+0000 7f8f7d071700 -1 mgr.server reply reply (22) Invalid argument 'exporter'
Updated by Dhairya Parmar 9 months ago
Venky Shankar wrote:
Dhairya shared the backtrace in cephfs slack channel
[...]
and everything else (including the backtrace) can be found in the mgr log that I shared.
Updated by Dhairya Parmar 9 months ago
export interface not working as well
sh-4.4$ ceph nfs export ls mynfs
Error EINVAL: Traceback (most recent call last):
File "/usr/share/ceph/mgr/mgr_module.py", line 1776, in _handle_command
return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
return self.func(mgr, **kwargs)
File "/usr/share/ceph/mgr/object_format.py", line 528, in _format_response
robj = f(*args, **kwargs)
File "/usr/share/ceph/mgr/nfs/module.py", line 97, in _cmd_nfs_export_ls
return self.export_mgr.list_exports(cluster_id=cluster_id, detailed=detailed)
File "/usr/share/ceph/mgr/nfs/export.py", line 440, in list_exports
self._validate_cluster_id(cluster_id)
File "/usr/share/ceph/mgr/nfs/export.py", line 369, in _validate_cluster_id
clusters = known_cluster_ids(self.mgr)
File "/usr/share/ceph/mgr/nfs/export.py", line 53, in known_cluster_ids
clusters = set(available_clusters(mgr))
File "/usr/share/ceph/mgr/nfs/utils.py", line 71, in available_clusters
orchestrator.raise_if_exception(completion)
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 228, in raise_if_exception
raise e
KeyError: 'exporter'