Project

General

Profile

Bug #46176

mgr/dashboard: NFS list timeout error

Added by Tatjana Dehler almost 4 years ago. Updated almost 2 years ago.

Status:
Closed
Priority:
High
Assignee:
-
Category:
Component - NFS
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

The NFS overview list is returning a timeout error if the related pool contains too many objects. The error seems to occur because of the following code lines: https://github.com/ceph/ceph/blob/nautilus/src/pybind/mgr/dashboard/services/ganesha.py#L795-L814 or https://github.com/ceph/ceph/blob/master/src/pybind/mgr/nfs/export.py#L295-L303
The for-loop needs to iterate all objects returned by `ioctx.list_objects()` which may take a lot of time if the pool contains many objects. Would it be possible to get the information in another way than iterating the whole list of objects?


Related issues

Related to Dashboard - Documentation #46530: mgr/dashboard: NFS ganesha object organization Resolved

History

#1 Updated by Tatjana Dehler almost 4 years ago

  • Description updated (diff)

#2 Updated by Tatjana Dehler over 3 years ago

If looping through all the objects in a pool (https://github.com/ceph/ceph/blob/master/src/pybind/mgr/dashboard/services/ganesha.py#L798-L817) is the real issue, there might be an idea for environments deployed by the Orchestrator:

If it's possible to get a list of the config files e.g. `conf-x` from the Orchestrator, then we might be able to get a list of the export names (export-y) from these files. We can then use the export name to look it up directly by calling the `Ioctx.read` function: https://docs.ceph.com/docs/master/rados/api/python/#rados.Ioctx.read . That way we can get the exports directly by name without looping through all objects of the pool.

#3 Updated by Kiefer Chang over 3 years ago

Thanks @Tatjana, the idea is good and we can avoid the object enumeration with it. Here are some problems we might encounter:

  • Users can create exports without associating any daemons. That is, `conf-<daemon>` object has no `%url ...` entry, but there still are some `export-<id>` objects. In this case, those exports become orphans and are not visible in the Dashboard.
  • How to get all daemons in order to resolve those `conf-<daemon>` objects if the Ceph cluster is deployed by downstream tools? Even we only support this after Octopus, we can't assume the cluster is deployed by the Orchestrator.
  • Dashboard can't manage daemons deployed outside the Orchestrator.

#4 Updated by Kiefer Chang over 3 years ago

In the comment of issue #46327 and some chats, CLI in `volumes` module is also suggested to be used by Dashboard.

In this case, we can assure there is only one way to manage NFS Ganesha exports.

The module provides an interface to manage NFS clusters. It:
- manipulates RADOS objects for exports like the Dashboard does.
- asks Orchestrator to create NFS service if it needs.
- Create FS volumes and subvolumes
- Currently it can't manage RGW exports. (Thanks Varsha for the info).

See the following links for more information.

[1] https://github.com/ceph/ceph/blob/0afbdca6932ca77aefac3dc3c2ee5e165429a8c9/src/pybind/mgr/volumes/module.py#L257-L334
[2] https://github.com/ceph/ceph/blob/master/doc/cephfs/fs-nfs-exports.rst
[3] https://pad.ceph.com/p/nfs

The implementation suffers the same problem if there are huge amounts of objects in a pool. Because it iterates all objects to get `conf-xxx` and `export-xxx` objects.

#5 Updated by Tatjana Dehler over 3 years ago

#6 Updated by Varsha Rao over 3 years ago

Kiefer Chang wrote:

In the comment of issue #46327 and some chats, CLI in `volumes` module is also suggested to be used by Dashboard.

In this case, we can assure there is only one way to manage NFS Ganesha exports.

The module provides an interface to manage NFS clusters. It:
- manipulates RADOS objects for exports like the Dashboard does.
- asks Orchestrator to create NFS service if it needs.
- Create FS volumes and subvolumes
- Currently it can't manage RGW exports. (Thanks Varsha for the info).

See the following links for more information.

[1] https://github.com/ceph/ceph/blob/0afbdca6932ca77aefac3dc3c2ee5e165429a8c9/src/pybind/mgr/volumes/module.py#L257-L334
[2] https://github.com/ceph/ceph/blob/master/doc/cephfs/fs-nfs-exports.rst
[3] https://pad.ceph.com/p/nfs

The implementation suffers the same problem if there are huge amounts of objects in a pool. Because it iterates all objects to get `conf-xxx` and `export-xxx` objects.

No, it does not as volume's nfs interface creates a separate pool for nfs-ganesha conf, export and grace objects. In the same pool cephfs or any other data is not saved. We assume not many exports will be created.

#7 Updated by Ernesto Puerta almost 3 years ago

  • Project changed from mgr to Dashboard
  • Category changed from 144 to Component - NFS

#8 Updated by Volker Theile almost 2 years ago

  • Description updated (diff)
  • Priority changed from Normal to High
  • Affected Versions v14.2.9, v18.0.0 added

#9 Updated by Ernesto Puerta almost 2 years ago

Is this still an issue in Pacific/Quincy NFS? It looks like related to the old NFS way. If no longer happening, can we close it?

#10 Updated by Tatjana Dehler almost 2 years ago

  • Status changed from New to Closed

Ernesto Puerta wrote:

Is this still an issue in Pacific/Quincy NFS? It looks like related to the old NFS way. If no longer happening, can we close it?

Oh! Good question. I can't tell if it's still an issue in Pacific or Quincy. I think we can close and re-open it if needed.

Also available in: Atom PDF