Project

General

Profile

Feature #47145

cephadm: Multiple daemons of the same service on single host

Added by Sebastian Wagner 8 months ago. Updated 11 days ago.

Status:
Closed
Priority:
Low
Assignee:
Category:
cephadm/scheduler
Target version:
-
% Done:

0%

Source:
Tags:
ceph-nano
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

ceph orch apply rgw myorg us-east-1 --placement="2 myhost1 myhost1" 

We use multiple replicas to make a service HA. Having them on the same host provides little benefit.
Exception is a ceph-nano style deployment where you need multiple MGRs to be able to upgrade them properly.
Adding this would be great, but low priority.


Related issues

Related to Orchestrator - Bug #44910: cephadm: PlacementSpec host1:192.168.0.2,host1:192.168.0.2 Rejected
Duplicated by Orchestrator - Feature #48114: Cephadm to support Adding multiple instances of RGW in same node for 5.0 release Duplicate
Blocked by Orchestrator - Feature #48822: Add proper port management to mgr/cephadm Closed
Blocks Orchestrator - Tasks #49490: cephadm additions/changes to support everything rgw.py needs New

History

#1 Updated by Sebastian Wagner 5 months ago

  • Related to Bug #44910: cephadm: PlacementSpec host1:192.168.0.2,host1:192.168.0.2 added

#2 Updated by Sebastian Wagner 5 months ago

  • Duplicated by Feature #48114: Cephadm to support Adding multiple instances of RGW in same node for 5.0 release added

#3 Updated by Sebastian Wagner 4 months ago

  • Tags set to ceph-nano

#4 Updated by Sebastian Wagner 2 months ago

  • Blocked by Feature #48822: Add proper port management to mgr/cephadm added

#5 Updated by Sebastian Wagner 2 months ago

in order to co-locate daemons, we have to use different ports for those new daemons.

#6 Updated by Sebastian Wagner about 2 months ago

service_type: rgw
service_id: realm.zone
placement:
  label: rgw
  count: 3
  allow-co-located: true

alternatively:

service_type: rgw
service_id: realm.zone
placement:
  hosts:
  - host1:1.2.3.0/24=name
  - host1:1.2.3.0/24=name
  - host2
  - host2

or

Edit: Doesn't work, as it breaks the possibility to name individual daemons

service_type: rgw
service_id: realm.zone
placement:
  hosts:
  - host1,count=8  # add "count" to host placement spec

or

Edit: IMO daemons-per-host is in conflict with count. And I don't see a clear use case except for rgw.py

service_type: rgw
service_id: realm.zone
placement:
  label: rgw
  count: 3
  daemons-per-host: 8 

or

Edit: IMO too complicated

placement:
  - label: rgw-big
    count-per-host: 8
    count: 24
  - label: rgw-small
    count: 2
  - hosts: host1
    names: a,b,c
    count-per-host: 3
  - label: foo
    count: 3

#7 Updated by Sebastian Wagner about 2 months ago

  • Blocks Tasks #49490: cephadm additions/changes to support everything rgw.py needs added

#8 Updated by Sebastian Wagner about 1 month ago

Kubernetes:

The below yaml snippet of the webserver deployment has podAntiAffinity and podAffinity configured. This informs the scheduler that all its replicas are to be co-located with pods that have selector label app=store. This will also ensure that each web-server replica does not co-locate on a single node.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
spec:
  selector:
    matchLabels:
      app: web-store
  replicas: 3
  template:
    metadata:
      labels:
        app: web-store
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - web-store
            topologyKey: "kubernetes.io/hostname" 
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - store
            topologyKey: "kubernetes.io/hostname" 
      containers:
      - name: web-app
        image: nginx:1.16-alpine

#9 Updated by Sebastian Wagner about 1 month ago

  • Status changed from New to Fix Under Review
  • Assignee set to Sage Weil
  • Pull request ID set to 39979

#10 Updated by Sebastian Wagner 11 days ago

  • Status changed from Fix Under Review to Closed

Also available in: Atom PDF