cephadm: Multiple daemons of the same service on single host
ceph orch apply rgw myorg us-east-1 --placement="2 myhost1 myhost1"
We use multiple replicas to make a service HA. Having them on the same host provides little benefit.
Exception is a ceph-nano style deployment where you need multiple MGRs to be able to upgrade them properly.
Adding this would be great, but low priority.
#6 Updated by Sebastian Wagner about 2 months ago
service_type: rgw service_id: realm.zone placement: label: rgw count: 3 allow-co-located: true
service_type: rgw service_id: realm.zone placement: hosts: - host1:22.214.171.124/24=name - host1:126.96.36.199/24=name - host2 - host2
Edit: Doesn't work, as it breaks the possibility to name individual daemons
service_type: rgw service_id: realm.zone placement: hosts: - host1,count=8 # add "count" to host placement spec
Edit: IMO daemons-per-host is in conflict with count. And I don't see a clear use case except for rgw.py
service_type: rgw service_id: realm.zone placement: label: rgw count: 3 daemons-per-host: 8
Edit: IMO too complicated
placement: - label: rgw-big count-per-host: 8 count: 24 - label: rgw-small count: 2 - hosts: host1 names: a,b,c count-per-host: 3 - label: foo count: 3
#8 Updated by Sebastian Wagner about 1 month ago
The below yaml snippet of the webserver deployment has podAntiAffinity and podAffinity configured. This informs the scheduler that all its replicas are to be co-located with pods that have selector label app=store. This will also ensure that each web-server replica does not co-locate on a single node.
apiVersion: apps/v1 kind: Deployment metadata: name: web-server spec: selector: matchLabels: app: web-store replicas: 3 template: metadata: labels: app: web-store spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - web-store topologyKey: "kubernetes.io/hostname" podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - store topologyKey: "kubernetes.io/hostname" containers: - name: web-app image: nginx:1.16-alpine