Project

General

Profile

Feature #44929

Add support DG_AFFINITY env var parsing.

Added by Joshua Schmid 8 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Development
Tags:
Backport:
octopus,nautilus, mimic
Reviewed:
Affected Versions:
Pull request ID:

Description

This is currently only relevant for cephadm.

We need a way to pass certain flags to ceph-volume that are not particularly useful to be exposed via ceph-volume's arg parser

  • drivegroup affinity

When creating a set of osds using the orchestrator (currently only cephadm) you will end up using OSDSpecs which inherits from DriveGroupSpec.
Since it's possible to apply multiple different drivegroups on one node we need a way to map osd daemons to their respective drivegroup persistently

The command would look something like this:

DG_AFFINITY='custom_name' ceph-volume lvm batch /dev/sda /dev/sdb

internally this would save the dg_affinity key with the value of "custom_name" in the tags of /dev/sda and /dev/sdb (their respective LV counterpart of course)

To retrieve the information we can leverage the existing `inventory` feature. This key will only be exposed when using the `--format $any_non_plain` flag as this information is almost always useless to the end user.

ceph-volume lvm inventory /dev/sda --format json-pretty
{
... # metadata
dg_affinity: 'custom_name'
... # more metadata

}

CONSIDERATIONS:

There is still the question if it makes sense to add proper arg parsing/passing for these flags to make it less awkward.

`ceph-volume lvm batch /dev/sda --dg-affinity $custom_name`

This seems more natural, but isn't very likely to be used by an actual person as drivegroups are typically leveraged by orchestration systems.

It's even more weird and awkward for the BLACKLISTED_DEVICES var env. If a user wants to exclude a certain disk from the command that he's running he can just exclude it from the ACTUAL command instead of passing it to ceph-volume as an ENV_VAR.

However, using the method described here gives us some advantages for orchestration systems.

If we'd do it like a user and just pass the adjusted command down to ceph-volume, we loose reporting functionality.

Let's assume we have two disks, /dev/sda, /dev/sdb and we want to blacklist /dev/sdb.

A user would do the following.


ceph-volume lvm batch /dev/sda 

The orchestrator should however do this:


BLACKLISTED_DEVICES=/dev/sdb ceph-volume lvm batch /dev/sda /dev/sdb

The reason for this is:

If you forgot (or for whatever reason) that you blacklisted a certain device, it's hard to debug and find out what's going on.
However, if you use the ENV_VAR method you can show that in the inventory output that cephadm also uses to show the device status.

BLACKLISTED_DEVICES=/dev/sdb ceph-volume inventory --format json-pretty

Will now print in the "rejected" field "blacklisted by user".


Related issues

Blocks RADOS - Bug #44755: Create stronger affinity between drivegroup specs and osd daemons Resolved
Copied to ceph-volume - Feature #45374: Add support for BLACKLISTED_DEVICES env var parsing. New
Copied to ceph-volume - Backport #45481: octopus: Add support DG_AFFINITY env var parsing. Resolved
Copied to ceph-volume - Backport #45482: mimic: Add support DG_AFFINITY env var parsing. Resolved
Copied to ceph-volume - Backport #45483: nautilus: Add support DG_AFFINITY env var parsing. Resolved

History

#1 Updated by Joshua Schmid 8 months ago

  • Blocks Bug #44755: Create stronger affinity between drivegroup specs and osd daemons added

#2 Updated by Jan Fajerski 8 months ago

Sounds all good to me.

Two things:
- Let's prepend the env var names with CEPH_VOLUME_. That fits the current convention.
- Do we want to add the blacklist as a glob maybe?

#3 Updated by Joshua Schmid 8 months ago

Jan Fajerski wrote:

Sounds all good to me.

Two things:
- Let's prepend the env var names with CEPH_VOLUME_. That fits the current convention.

agreed

- Do we want to add the blacklist as a glob maybe?

sure, we can do that

#5 Updated by Jan Fajerski 7 months ago

  • Copied to Feature #45374: Add support for BLACKLISTED_DEVICES env var parsing. added

#6 Updated by Jan Fajerski 7 months ago

  • Subject changed from Add support for BLACKLISTED_DEVICES and DG_AFFINITY env var parsing. to Add support DG_AFFINITY env var parsing.
  • Description updated (diff)
  • Status changed from New to Pending Backport
  • Backport set to octopus,nautilus, mimic
  • Pull request ID set to 34436

#7 Updated by Nathan Cutler 7 months ago

  • Copied to Backport #45481: octopus: Add support DG_AFFINITY env var parsing. added

#8 Updated by Nathan Cutler 7 months ago

  • Copied to Backport #45482: mimic: Add support DG_AFFINITY env var parsing. added

#9 Updated by Nathan Cutler 7 months ago

  • Copied to Backport #45483: nautilus: Add support DG_AFFINITY env var parsing. added

#10 Updated by Jan Fajerski 4 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF