Feature #55663
closed
cephadm/nfs: enable cephadm to provide one virtual per ganesha instance of the NFS service
Added by Ramana Raja almost 2 years ago.
Updated 11 months ago.
Description
cephadm allows ingress service (haporxy+keepalived) to be deployed in front of NFS-ganesha service (a cluster of NFS-ganesha daemons) to provide HA stable virtual IP to NFS clients, https://docs.ceph.com/en/quincy/mgr/nfs/#ingress . One of the downsides of this HA NFS service setup is that the backend NFS servers cannot see the source client IPs, and can only see the proxy server's IP. So use-cases such as OpenStack manila, where the NFS-ganesha enforces client IP based authorization for export access cannot be met.
A HA model for NFS service was discussed earlier in https://pad.ceph.com/p/cephadm-nfs-ha (Option 2 and 2a) and https://www.spinics.net/lists/dev-ceph/msg03442.html (Option 5 and 6), where cephadm internally manages the virtual IP of ganesha servers. cephadm would set up one virtual IP per each ganesha daemon it deploys. It would add or remove the virtual IP when creating or removing a ganesha server.
From https://www.spinics.net/lists/dev-ceph/msg03442.html,
"
single ganesha + single virtual IP
- 1 ganesha daemon
- 1 virtual IP that follows the ganesha daemon
- on failure, cephadm would deploy ganesha elsewhere + move virtual IP
- not implemented
multiple ganesha + multiple virtual IPs
- N ganesha daemons
- N virtual IPs
- requires ganesha changes to (1) make ganesha aware of peers and (2)
instruct clients to move around
- on failure, cephadm would deploy failed ganesha elsewhere + move
that virtual IP
- not implemented (in cephadm or ganesha)
"
- Subject changed from cephadm/nfs: enable cephadm to internally manage the virtual IPs of ganesha daemons to cephadm/nfs: enable cephadm to provide one virtual per ganesha instance of the NFS service
- Description updated (diff)
- Description updated (diff)
- Related to Bug #54470: Can't mount NFS exports with IP access restrictions when the Ingress daemon is deployed in front of the NFS service added
- Related to Bug #54470: Can't mount NFS exports with IP access restrictions when the Ingress daemon is deployed in front of the NFS service added
- Related to deleted (Bug #54470: Can't mount NFS exports with IP access restrictions when the Ingress daemon is deployed in front of the NFS service)
Reportedly, the PROXY protocol has options to preserve the client's IP address. HAProxy should support this. Is this something that we have found doesn't work for NFS-Ganesha for any reason? Or perhaps we haven't been aware of this option yet?
Blaine Gardner wrote:
Reportedly, the PROXY protocol has options to preserve the client's IP address. HAProxy should support this. Is this something that we have found doesn't work for NFS-Ganesha for any reason? Or perhaps we haven't been aware of this option yet?
We discussed this in the Ceph orchestrator's meeting, https://pad.ceph.com/p/orchestration-weekly (See notes for May 10). The issue was that the NFS-Ganesha currently doesn't support PROXY protocol. Discussions with NFS-Ganesha core developers about implementing PROXY protocol is in very early stage. We'll have to wait on how those discussions proceed.
Using haproxy in front of NFS server has its disadvantages:
- haproxy gets in the way of moving NFS clients to a different NFS server when rebalancing of live-shrinking in the NFS cluster.
See https://www.spinics.net/lists/dev-ceph/msg03431.html for more details
- it is less performant as haproxy introduces an extra hop
Implementing this feature is orthogonal to introducing improvements in ingress(haproxy/keepalived) + nfs solution.
- Pull request ID set to 47199
- Status changed from New to In Progress
- Assignee set to Adam King
- Status changed from In Progress to Pending Backport
- Backport set to reef, quincy
- Copied to Backport #59630: quincy: cephadm/nfs: enable cephadm to provide one virtual per ganesha instance of the NFS service added
- Copied to Backport #59631: reef: cephadm/nfs: enable cephadm to provide one virtual per ganesha instance of the NFS service added
- Tags set to backport_processed
- Status changed from Pending Backport to Resolved
Also available in: Atom
PDF