Project

General

Profile

Feature #44556

cephadm: preview drivegroups

Added by Joshua Schmid 21 days ago. Updated 2 days ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
cephadm
Target version:
% Done:

0%

Source:
Development
Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

The osd deployment in cephadm happens async in the background.

When using drivegroups, it may be not always clear what will happen since the outcome is dependent on the inventory of a node/cluster.

I.e. a very simple drivegroup

data_devices:
  all: True

Is probably not worth previewing. However we allow for more complex configurations where the outcome is not always clear.
It's also possible that the hardware configuration of a node has changed and therefore the drivegroups will take an unwanted/unexpected effect.
To prevent this we should implement feature $subject.

There is at least one thing we have to consider in order to make this work:

  • Since we’re running (not only) osd deployments continuously in the background there’s no real sense in showing a preview since it will be applied anyways within the next few minutes.

To tackle that we should either consider recommending drivegroups that are more complex always with the "unmanaged" flag or add some similar mechanism.

Implementation:

ceph-volume actually provides the necessary data already

Note the --report flag

ceph-volume lvm batch /dev/sdx, /dev/sdy --report --format json

where sdx is available and sdy is not

returns something like this:

{
    "changed": true,
    "osds": [
        {
            "block.db": {},
            "data": {
                "human_readable_size": "24.00 GB",
                "parts": 1,
                "path": "/dev/sdx",
                "percentage": 100.0,
                "size": 24
            }
        }
    ],
    "vgs": []
}

if none of the disks are available, c-v returns an empty dict.

This data just needs to be collected from the nodes and aggregated in cephadm.

CLI:

there are multiple options - Here's a list of commands that came to my mind

  • `ceph orch drivegroup preview`
  • `ceph orch osd drivegroup preview`
  • `ceph orch osd apply <-i|--all-available-devices> --preview`

The last command has the advantage that you can adjust your drivegroup, save it to the mon_store and get direct feedback.

Edit: However, this is an intrusion in the apply mechanism, which I'd like to keep as independent as possible.
Currently I'm in favor of #)1 oder #)2


Related issues

Related to mgr - Bug #44808: mgr/dashboard: Allow users to specify an unmanaged ServiceSpec when creating OSDs New

History

#1 Updated by Sebastian Wagner 20 days ago

  • Source changed from Community (dev) to Development

#2 Updated by Joshua Schmid 7 days ago

  • Description updated (diff)

#3 Updated by Joshua Schmid 7 days ago

  • Description updated (diff)

#4 Updated by Joshua Schmid 6 days ago

  • Status changed from New to In Progress
  • Assignee set to Joshua Schmid

#5 Updated by Joshua Schmid 6 days ago

This is the output of:

ceph orch drivegroups preview --format yaml

for drivegroup

service_type: osd
service_id: test_dg
data_devices:
  size: 25G
db_devices:
  size: 50G
placement:
  host_pattern: cephadm-dev
unmanaged: true

data:
  changed: true
  osds:
  - block.db:
      human_readable_size: 16.33 GB
      path: 'vg: vg/lv'
      percentage: 33
      size: 17537783125
    data:
      human_readable_size: 24.00 GB
      path: /dev/vdb
      percentage: 100.0
      size: 25769803776.0
  - block.db:
      human_readable_size: 16.33 GB
      path: 'vg: vg/lv'
      percentage: 33
      size: 17537783125
    data:
      human_readable_size: 24.00 GB
      path: /dev/vdi
      percentage: 100.0
      size: 25769803776.0
  - block.db:
      human_readable_size: 16.33 GB
      path: 'vg: vg/lv'
      percentage: 33
      size: 17537783125
    data:
      human_readable_size: 24.00 GB
      path: /dev/vdj
      percentage: 100.0
      size: 25769803776.0
  vg:
    devices: /dev/vdl
    human_readable_size: 49.00 GB
    human_readable_sizes: 16.33 GB
    parts: 3
    percentages: 33
    size: 52613349376
    sizes: 17537783125
  vgs: []
drivegroup: test_dg
host: cephadm-dev

or with

ceph orch drivegroups preview

in table form

NAME    HOST        DATA     DB       WAL 
test_dg cephadm-dev /dev/vdb /dev/vdl -   
test_dg cephadm-dev /dev/vdi /dev/vdl -   
test_dg cephadm-dev /dev/vdj /dev/vdl - 

#6 Updated by Joshua Schmid 6 days ago

  • Description updated (diff)

#7 Updated by Joshua Schmid 6 days ago

  • Description updated (diff)

#8 Updated by Kiefer Chang 5 days ago

  • Pull request ID set to 34216

#9 Updated by Kiefer Chang 5 days ago

  • Pull request ID deleted (34216)

#10 Updated by Joshua Schmid 5 days ago

  • Pull request ID set to 34216

#11 Updated by Joshua Schmid 5 days ago

When we want to have preview functionality for other components aswell, we should generalize the CLI a bit more.

maybe:

`ceph orch preview $component <optional service_id>`

The reason why I dislike:

ceph orch apply $component -i <file> --dry-run

is that you have to always supply it with a file or a struct in order to get a preview. If you already have applied a spec but just added new hardware (drives, nodes) you also want to preview the results without applying a new spec.

#12 Updated by Joshua Schmid 2 days ago

I guess we want to have the following features:

1) preview a service if the spec is already applied (useful when hardware changes, like a replacement operation or node additions)
2) preview a service if the spec is not yet applied (useful to see what would happen when applying the spec)

While the `ceph orch apply $service` command seems to be right place for 2) it kinda doesn't make sense for 1).

If we however introduce a new command like the one suggested above(`ceph orch preview $component`, will have a disconnect between applying and previewing specs.

If we really want to have preview capabilities for services we should incorporate this into the core command. So what about re-modeling it like so:

Add|Preview a set of specs with a file

ceph orch spec -i --apply|--preview

Add|Preview a specific spec without a file:

ceph orch spec $service <spec_with_strings> --apply|--preview

Preview a specific "stored" spec:

ceph orch spec osd.example_spec  --preview

Preview all "stored" specs:

ceph orch spec --preview

This would also allow to put operations on the spec in the right place. For example

ceph orch spec osd.example_spec --inspect|--dump|--export

I see that this is a change to the CLI yet again, but let me know what you think..

#13 Updated by Sebastian Wagner 2 days ago

I guess we want to have the following features:

1) preview a service if the spec is already applied (useful when hardware changes, like a replacement operation or node additions)
2) preview a service if the spec is not yet applied (useful to see what would happen when applying the spec)

While the `ceph orch apply $service` command seems to be right place for 2) it kinda doesn't make sense for 1).

Right, but I think we can get away without introducing any new concepts to the CLI.

If we however introduce a new command like the one suggested above(`ceph orch preview $component`, will have a disconnect between applying and previewing specs.

If we really want to have preview capabilities for services we should incorporate this into the core command. So what about re-modeling it like so:

< pre> #haha, this breaks Redmine's markup
Add|Preview a set of specs with a file

ceph orch spec -i --apply|--preview

ceph orch apply -i --dry-run

really looks cleaner to me. We should IMO limit the number concepts users need to understand and by adding --dry-run, we don't need to introduce a whole new

ceph orch spec

prefix.

Add|Preview a specific spec without a file:

ceph orch spec $service <spec_with_strings> --apply|--preview

What do you have in mind for spec_with_strings ?

Preview a specific "stored" spec:

ceph orch spec osd.example_spec --preview

Isn't this a dry-run when for unmanaged to False? Something like

ceph orch apply --service_type osd --service_id my-osd-spec --unmanaged=False

?

Preview all "stored" specs:

ceph orch spec --preview

Hm, do we really need this? Looks a bit scary to set unmanaged=False for all specs at once. you can always do something like

ceph orch ls --export | grep -v "unmanaged: false" | ceph orch apply -i -

Which is imo scary enough.

This would also allow to put operations on the spec in the right place. For example

ceph orch spec osd.example_spec --inspect|--dump|--export

ceph orch ls --export --service_name osd.example_spec

what do you have in mind for --inspect or --dump?

I see that this is a change to the CLI yet again, but let me know what you think..

Actually it looks to me as if we don't need to add anything new, except for a --dry-run!

#14 Updated by Sebastian Wagner 2 days ago

  • Related to Bug #44808: mgr/dashboard: Allow users to specify an unmanaged ServiceSpec when creating OSDs added

#15 Updated by Joshua Schmid 2 days ago

Shortened this discussion with a f2t talk with Sebastian. Here are the results

Keep the old syntax but expand with the following features:

  • Allow to target existing(stored) specs.
ceph orch apply --service-id $service_id 
  • Allow to change the 'unmanaged' flag for existing specs
ceph orch apply --service-id $service_id --unmanaged true|false
  • add a --dry-run|--preview flag to apply
ceph orch apply -i|--service-id|<host spec via strings> --dry-run|--preview
  • generate and output a preview after any `apply` operation.

This way, the user will get direct feedback of what will happen.

Also available in: Atom PDF