Ceph » Orchestrator

The current HAProxy based ingress option helps to make the Ceph
object service highly available, but is not scale out in terms
of it's ability to handle more traffic than can otherwise be
handled by a single HAproxy host.

Proposed is to include a 'strategy' parameter in an ingress service, that
controls whether the current HAproxy scheme is deployed, or whether a new
IPVS based direct routing scheme is used instead.

The new scheme will still rely on using keepalived to manage the VIP, but
instead of using HAProxy on each host, we will configure IPVS with direct
routing. Direct routing works by rewritting the destination mac address
on packets from the client to one of the other balancers. The balancers
return traffic directly back to the client. The result is the ability to
scale GET traffic (LB to client tx). Creation of Foo-over-UDP tunnels can
facilitate sending the rewritten frames to hosts on different subnets,
which is common in modern spine/leaf networks that terminate L2 domains
at the top of rack and use routing between racks.

L4 / IPVS Ingress for cephadm

 ------------    ------------    ------------    ------------  
|   CLIENT   |  |   CLIENT   |  |  CLIENT    |  |   CLIENT   | 
 ------------    ------------    ------------    ------------  
  ^ |               /  ^            / ^             / ^        
  | |              /   |           /  |            /  |        
  | |             /    |          /   |           /   |        
  | |            /     |         /    |          /    |        
  | | ----------       |        /     |         /     |        
  | | | ---------------)--------      |        /      |        
  | | | |  ------------)--------------)--------       |        
  | | | | |            |              |               |        
  V V V V V            |tx            |tx             |tx      
 ------------    ------------    ------------    ------------  
|     IP     |  |     IP     |  |     IP     |  |     IP     | 
|    VIP     |  |            |  |            |  |            | 
|            |  |            |  |            |  |            | 
 ------------    ------------    ------------    ------------  
    | | |              ^               ^              ^        
    | | |              |               |              |        
    | |  --------------                |              |        
    |  ---------------------------------              |        
     --------------------------------------------------       

More details

https://blog.cloudflare.com/high-availability-load-balancers-with-maglev/

service_type: ingress
service_id: rgw.something             # adjust to match your existing RGW service
placement:
  hosts:
    - host1
    - host2
    - host3spec:
  backend_service: rgw.something      # adjust to match your existing RGW service
  strategy: [ L7 | L4 ]               # NEW: L7 for haproxy, L4 for IPVS
  virtual_ip: <string>/<string>       # ex: 192.168.20.1/24
  # L2 / IPVS options

  # L7 / HAProxy options
  frontend_port: <integer>            # ex: 8080
  monitor_port: <integer>             # ex: 1967, used by haproxy for load balancer status
  virtual_interface_networks: [ ... ] # optional: list of CIDR networks
  ssl_cert: |                         # optional: SSL certificate and key
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
    -----BEGIN PRIVATE KEY-----
    ...
    -----END PRIVATE KEY-----