Ceph » mgr » Dashboard

Description of problem¶

Fail to create a NFS export with CephFS FSAL in a Cephadm cluster.

How reproducible¶

• Bootstrap a cephadm cluster (1 MON + 1 MGR).
• Create some OSDs
• Create a CephFS and at least one MDSs. (ceph fs volume create)
• Create a pool to store Ganesha configs
• Deploy a Cephadm NFS service with that pool
• Use the dashboard to create an NFS export with a non-existing directory in CephFS, an error is displayed:
  

• The exception in mgr log:

debug 2020-12-21T09:11:23.328+0000 7f9ed7b20700  0 [dashboard ERROR taskexec] Error while calling Task(ns=nfs/create, md={'path': '/abc', 'fsal': 'CEPH', 'cluster_id': 'foo'})
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/tools.py", line 559, in _run
    val = self.task.fn(*self.task.fn_args, **self.task.fn_kwargs)  # type: ignore
  File "/usr/share/ceph/mgr/dashboard/controllers/nfsganesha.py", line 153, in create
    'clients': clients
  File "/usr/share/ceph/mgr/dashboard/services/ganesha.py", line 972, in create_export
    self._save_export(export)
  File "/usr/share/ceph/mgr/dashboard/services/ganesha.py", line 952, in _save_export
    export.fsal.create_path(export.path)
  File "/usr/share/ceph/mgr/dashboard/services/ganesha.py", line 493, in create_path
    cfs.mk_dirs(path)
  File "/usr/share/ceph/mgr/dashboard/services/cephfs.py", line 152, in mk_dirs
    self.cfs.mkdirs(path, 0o755)
  File "cephfs.pyx", line 997, in cephfs.LibCephFS.mkdirs
cephfs.Error: error in mkdirs /abc: Permission denied [Errno 13]

Additional info¶

One thing worth mentioning is if I reload the Dashboard module (or enable another module like Prometheus), the export can be created.
The Dashboard uses LibCephFs with the `g_ceph_context` context to create folders in a CephFS.

• https://github.com/ceph/ceph/blob/851d10ed3516a88586515913d95df514d065e53d/src/pybind/mgr/dashboard/services/cephfs.py#L41
• https://github.com/ceph/ceph/blob/851d10ed3516a88586515913d95df514d065e53d/src/pybind/mgr/mgr_module.py#L1517
• https://github.com/ceph/ceph/blob/851d10ed3516a88586515913d95df514d065e53d/src/mgr/ActivePyModules.cc#L863

Somehow the context doesn't have enough permission in this case.

I created a simple Vagrantfile that can reproduce this problem:

# Install vagrant and vagrant-libvirt first.

mkdir test-export
cd test-export
wget https://raw.githubusercontent.com/bk201/ceph-resources/master/issues/mgr-context/Vagrantfile
vagrant up

# Bootstrap a simple cephadm cluster
vagrant ssh
sudo -i
/scripts/bootstrap.sh

# Create OSDs, a pool, and an NFS service.

cephadm shell -m /scripts/

[ceph: root@storage0 /]# /mnt/setup.sh

# Wait until nfs service `nfs.foo.storage0` is ready:
[ceph: root@storage0 /]# ceph orch ps
NAME                   HOST      STATUS          REFRESHED  AGE   VERSION  IMAGE NAME               IMAGE ID      CONTAINER ID
crash.storage0         storage0  running (84s)   17s ago    84s   15.2.8   docker.io/ceph/ceph:v15  5553b0cb212c  eacae6580ef0
mds.a.storage0.siiihh  storage0  running (25s)   17s ago    24s   15.2.8   docker.io/ceph/ceph:v15  5553b0cb212c  ce633812c561
mgr.storage0.uigkyf    storage0  running (116s)  17s ago    116s  15.2.8   docker.io/ceph/ceph:v15  5553b0cb212c  73ac0376b19f
mon.storage0           storage0  running (117s)  17s ago    119s  15.2.8   docker.io/ceph/ceph:v15  5553b0cb212c  c73376ceaa0f
nfs.foo.storage0       storage0  running (23s)   17s ago    23s   3.3      docker.io/ceph/ceph:v15  5553b0cb212c  f4514fb98f5b
osd.0                  storage0  running (34s)   17s ago    34s   15.2.8   docker.io/ceph/ceph:v15  5553b0cb212c  fcdaa1174e45
osd.1                  storage0  running (32s)   17s ago    32s   15.2.8   docker.io/ceph/ceph:v15  5553b0cb212c  697d9f3feef6
osd.2                  storage0  running (30s)   17s ago    30s   15.2.8   docker.io/ceph/ceph:v15  5553b0cb212c  09ffdcc2ec68

# A script to help creating an export via the Dashboard API
[ceph: root@storage0 /]# /mnt/create_export.sh
{"detail": "error in mkdirs /abc: Permission denied [Errno 13]", "component": null, "status": 500, "task": {"name": "nfs/create", "metadata": {"path": "/abc", "fsal": "CEPH", "cluster_id": "foo"}}}

# The operation is succeeded if we reload the Dashboard module
[ceph: root@storage0 /]# /mnt/reload_dashboard.sh
[ceph: root@storage0 /]# /mnt/create_export.sh
{"name": "nfs/create", "metadata": {"path": "/abc", "fsal": "CEPH", "cluster_id": "foo"}}
[ceph: root@storage0 /]# /mnt/ls_dir.sh
[
  {
    "name": "abc",
    "path": "/abc",
    "parent": "/",
    "snapshots": [],
    "quotas": {
      "max_bytes": 0,
      "max_files": 0
    }
  }
]