Project

General

Profile

Feature #37083

ceph-volume should report device_id as part of the device list

Added by Anonymous over 5 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

When listing the device list, the orchestrator needs to get the device_id as defined in "ceph device" command.

History

#1 Updated by Alfredo Deza over 5 years ago

Could you expand a bit on this ticket? I've never seen `ceph device` before. Is that Nautilus only? When you say "listing the device list" what do you mean? listing from where?

#2 Updated by Anonymous over 5 years ago

Yeah, that's starting nautilus. A typical output looks like :

$ ceph device ls
DEVICE HOST:DEV DAEMONS LIFE EXPECTANCY
Crucial_CT1024M550SSD1_14160C164100 stud:sdd osd.40 >6w
Crucial_CT1024M550SSD1_14210C25B79E eutow:sds osd.19 >6w

Sorry, I was speaking about the inventory command not "listing the device". Sent the ticket too fast ;)

#3 Updated by Anonymous over 5 years ago

As we can't rely on a running ceph cluster we'll sadly have to duplicate the code for this feature.
Code is here :
https://github.com/ceph/ceph/blob/master/src/common/blkdev.cc#L395

#4 Updated by Alfredo Deza over 5 years ago

Seems like by 'ID' you mean something that gets composed somehow if certain information is not available. Do you need the exact same implementation from C++ in Python? There is a bunch
of mangling going on in there, and I worry that whatever we can come up with will not be a 100% match to what the C++ version does

#5 Updated by Anonymous over 5 years ago

We need to share the same format so the manager can 'recognize' as disk c-v exposes as the same disk as the one it have in its 'ceph device' list.

I'm not sure to get what part is difficult here as we already have the same information on our side.

#6 Updated by Alfredo Deza over 5 years ago

It is not clear to me what spec I should adhere to. It seems like it isn't 1 piece of information added, but to compose one from different items, mangle them in a specific way, and produce
a final identifier. If you could define that spec so that we can produce it to those standards that would be great.

#7 Updated by Jan Fajerski over 5 years ago

  • Assignee set to Jan Fajerski

I had a look into this and I have two discussion points:

The current c++ implementation uses the ID_SERIAL udev property and falls back to using /sys/block/$dev/model and the serial retrieved via the ioctl sys call.
I checked a few systems and ID_SERIAL seems to be somewhat unreliable (returning something different then $MODEL_$SERIAL). I think it would be more reliable to use ID_MODEL + _ + ID_SERIAL_SHORT. As an example I have a machine where ID_SERIAL=36b82a720ce6af0001b3d310f123b3f52.

One way to resolve this double implementation is to implement this in ceph-volume and have ceph call, say ceph-volume inventory and get the device_id from there. I think we can rely on ceph-volume being installed on every node?

#8 Updated by Anonymous over 5 years ago

Yes that would be better to get a single implementation to avoid mismatches.

I see two set of calls of get_device_id().
- One in the osd/OSD.cc so that's covered by the dependency from ceph-volume & ceph-osd.
- One in the mon/Monitor.cc. ceph-mon package doesn't depend on ceph-volume yet.

Sage made this code, would worth asking him if he would agree making a ceph-volume call to gain this feature.

My argument here, is as long a we a considering ceph-volume to be installed before ceph is ready (for the inventory), it wouldn't be too hard to add a dep between ceph-mon & ceph-volume. If we do in this direction, that would imply a splitted ceph-volume package for sure which have a dependcy relation with ceph-osd & ceph-mon.

#9 Updated by Alfredo Deza over 5 years ago

There is an assumption here that ceph-volume can be installed without any other ceph component. A lot of the internal machinery in ceph-volume requires Ceph to be installed and configured properly.

It is a problem to start pushing configuration management tasks into a tool that is really meant to provision OSDs. Although the inventory is already there, trying to make it work in situations where the system is not ready to provision an OSD will be a continuous source of pain.

We shouldn't consider ceph-volume at all for this use case.

#10 Updated by Anonymous over 5 years ago

@alfredo : We have no choice here. We need to list the devices before Ceph runs. So c-v have to be resilient when ceph isn't avail and reports what it founds on the existing devices.

#11 Updated by Alfredo Deza over 5 years ago

The choice is to have these configuration-management tasks be done with a configuration management tool, not in ceph-volume that is meant to provision OSDs.

I'm going to push back on items that are not aligned for provisioning an OSD, and not having Ceph installed and ready for an OSD certainly falls into a place where we don't want to go.

#12 Updated by Anonymous over 5 years ago

Alfredo Deza wrote:

The choice is to have these configuration-management tasks be done with a configuration management tool, not in ceph-volume that is meant to provision OSDs.

Ceph-volume is not only to provision OSDs but also list devices as an helper for orchestrator/manager. That's part of the orchestrator sandwich. Jan made a good PR on this list.

I'm going to push back on items that are not aligned for provisioning an OSD, and not having Ceph installed and ready for an OSD certainly falls into a place where we don't want to go.

During the Berlin meeting (on the orchestrator topic), something like 10 people stated this feature was necessary and agreed the location in the ceph-volume. So I wonder who is the 'we' you use.

The orchestrator sandwich needs the ceph-volume command to list the devices being free to use before ceph is deployed. If the actual design of c-v doesn't allow it, that means a patch will be needed. If you foresee a different implementation & design feel free to share it so it can be debated.

#13 Updated by Anonymous over 5 years ago

As per the discussion on 28th November, the device id implementation must remain in c++ code and in ceph-volume separetly and supports linux & BSD.
This four implementations must provide the same output for the same disk.

#14 Updated by Jan Fajerski over 5 years ago

  • Status changed from New to Resolved

Also available in: Atom PDF