Feature #55663
closedcephadm/nfs: enable cephadm to provide one virtual per ganesha instance of the NFS service
0%
Description
cephadm allows ingress service (haporxy+keepalived) to be deployed in front of NFS-ganesha service (a cluster of NFS-ganesha daemons) to provide HA stable virtual IP to NFS clients, https://docs.ceph.com/en/quincy/mgr/nfs/#ingress . One of the downsides of this HA NFS service setup is that the backend NFS servers cannot see the source client IPs, and can only see the proxy server's IP. So use-cases such as OpenStack manila, where the NFS-ganesha enforces client IP based authorization for export access cannot be met.
A HA model for NFS service was discussed earlier in https://pad.ceph.com/p/cephadm-nfs-ha (Option 2 and 2a) and https://www.spinics.net/lists/dev-ceph/msg03442.html (Option 5 and 6), where cephadm internally manages the virtual IP of ganesha servers. cephadm would set up one virtual IP per each ganesha daemon it deploys. It would add or remove the virtual IP when creating or removing a ganesha server.
From https://www.spinics.net/lists/dev-ceph/msg03442.html,
"
single ganesha + single virtual IP
- 1 ganesha daemon
- 1 virtual IP that follows the ganesha daemon
- on failure, cephadm would deploy ganesha elsewhere + move virtual IP
- not implemented
multiple ganesha + multiple virtual IPs
- N ganesha daemons
- N virtual IPs
- requires ganesha changes to (1) make ganesha aware of peers and (2)
instruct clients to move around
- on failure, cephadm would deploy failed ganesha elsewhere + move
that virtual IP
- not implemented (in cephadm or ganesha)
"
Updated by Ramana Raja almost 2 years ago
- Subject changed from cephadm/nfs: enable cephadm to internally manage the virtual IPs of ganesha daemons to cephadm/nfs: enable cephadm to provide one virtual per ganesha instance of the NFS service
Updated by Ramana Raja almost 2 years ago
- Related to Bug #54470: Can't mount NFS exports with IP access restrictions when the Ingress daemon is deployed in front of the NFS service added
Updated by Ramana Raja almost 2 years ago
- Related to Bug #54470: Can't mount NFS exports with IP access restrictions when the Ingress daemon is deployed in front of the NFS service added
Updated by Ramana Raja almost 2 years ago
- Related to deleted (Bug #54470: Can't mount NFS exports with IP access restrictions when the Ingress daemon is deployed in front of the NFS service)
Updated by Blaine Gardner almost 2 years ago
Reportedly, the PROXY protocol has options to preserve the client's IP address. HAProxy should support this. Is this something that we have found doesn't work for NFS-Ganesha for any reason? Or perhaps we haven't been aware of this option yet?
Updated by Ramana Raja almost 2 years ago
Blaine Gardner wrote:
Reportedly, the PROXY protocol has options to preserve the client's IP address. HAProxy should support this. Is this something that we have found doesn't work for NFS-Ganesha for any reason? Or perhaps we haven't been aware of this option yet?
We discussed this in the Ceph orchestrator's meeting, https://pad.ceph.com/p/orchestration-weekly (See notes for May 10). The issue was that the NFS-Ganesha currently doesn't support PROXY protocol. Discussions with NFS-Ganesha core developers about implementing PROXY protocol is in very early stage. We'll have to wait on how those discussions proceed.
Using haproxy in front of NFS server has its disadvantages:
- haproxy gets in the way of moving NFS clients to a different NFS server when rebalancing of live-shrinking in the NFS cluster.
See https://www.spinics.net/lists/dev-ceph/msg03431.html for more details
- it is less performant as haproxy introduces an extra hop
Implementing this feature is orthogonal to introducing improvements in ingress(haproxy/keepalived) + nfs solution.
Updated by Francesco Pantano almost 2 years ago
HA/NFS POC -> https://docs.google.com/document/d/19G_SBuVXu2IVTz3hWkbHs__vRF1bbAhnfkL3YSNkVkQ/edit
and a code example that deploy keepalived only:
https://github.com/fmount/ceph/commit/a2b2c36f0cf2dfd6bffa493ab55bebe032a7193c
Updated by Redouane Kachach Elhichou over 1 year ago
- Pull request ID set to 47199
Updated by Redouane Kachach Elhichou over 1 year ago
- Status changed from New to In Progress
- Assignee set to Adam King
Updated by Backport Bot 12 months ago
- Copied to Backport #59630: quincy: cephadm/nfs: enable cephadm to provide one virtual per ganesha instance of the NFS service added
Updated by Backport Bot 12 months ago
- Copied to Backport #59631: reef: cephadm/nfs: enable cephadm to provide one virtual per ganesha instance of the NFS service added