ceph-volume lvm create fails to detect multipath devices
My esteemed colleague and I are migrating our existing Ceph cluster to Luminous, and are rebuilding some of our OSDs to use BlueStore. We are using
ceph-volume lvm create to re-create a recently destroyed OSD on a disk we access using multipath. The device is reported by
lsblk as follows:
[root@ceph-disk-1 site-packages]# lsblk /dev/mapper/bay7 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT bay7 253:5 0 7.3T 0 mpath
Note that the type is
disk. This creates the following output when trying to create a BlueStore-backed OSD using that storage.
[root@ceph-disk-1 ~]# ceph-volume lvm create --bluestore --data /dev/mapper/bay7 --block.db /dev/nvme1n1p2 --osd-id 7 Running command: ceph-authtool --gen-print-key Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 963eec50-5d44-44b0-9b7d-e9be45b44b52 7 --> Was unable to complete a new OSD, will rollback changes --> OSD will be destroyed, keeping the ID because it was provided with --osd-id Running command: ceph osd destroy osd.7 --yes-i-really-mean-it stderr: destroyed osd.7 --> RuntimeError: Cannot use device (/dev/mapper/bay7). A vg/lv path or an existing device is needed
Through some Python debugging, we were able to find that
util.is_device('/dev/mapper/bay7') disagrees that this
mpath device is a usable device. On
master as of this writing, the culpable code is here: https://github.com/ceph/ceph/blob/5fee2689c87dea23cab10fee2a758f4d9a5b4959/src/ceph-volume/ceph_volume/util/disk.py#L204-L206
# ... src/ceph-volume/util/disk.py line 204-206 TYPE = lsblk(dev).get('TYPE') if TYPE: return TYPE == 'disk'
As reported by the
lsblk function defined earlier in the same file,
mpath. This matches what
lsblk reported earlier, but doesn't reveal that the
mpath is actually fronting a disk. We were able to work around this issue for the time being by amending the above code to be
# ... PATCHED src/ceph-volume/util/disk.py line 204-206 TYPE = lsblk(dev).get('TYPE') if TYPE: return TYPE == 'disk' or TYPE == 'mpath'
but I suspect this is likely not the ideal solution. After making this change, however, we re-ran our initial command, and it worked perfectly.
[ceph-deploy@ceph-disk-1 ~]$ sudo ceph-volume lvm create --bluestore --data /dev/mapper/bay7 --block.db /dev/nvme1n1p2 --osd-id 7 Running command: ceph-authtool --gen-print-key Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e2b78628-53f5-4a78-870c-990d2063305a 7 Running command: vgcreate --force --yes ceph-22c44762-565e-46b8-89aa-d379b6d42b2c /dev/mapper/bay7 stdout: Physical volume "/dev/mapper/bay7" successfully created. stdout: Volume group "ceph-22c44762-565e-46b8-89aa-d379b6d42b2c" successfully created Running command: lvcreate --yes -l 100%FREE -n osd-block-e2b78628-53f5-4a78-870c-990d2063305a ceph-22c44762-565e-46b8-89aa-d379b6d42b2c stdout: Logical volume "osd-block-e2b78628-53f5-4a78-870c-990d2063305a" created. Running command: ceph-authtool --gen-print-key Running command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-7 Running command: chown -R ceph:ceph /dev/dm-18 Running command: ln -s /dev/ceph-22c44762-565e-46b8-89aa-d379b6d42b2c/osd-block-e2b78628-53f5-4a78-870c-990d2063305a /var/lib/ceph/osd/ceph-7/block Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-7/activate.monmap stderr: got monmap epoch 3 Running command: ceph-authtool /var/lib/ceph/osd/ceph-7/keyring --create-keyring --name osd.7 --add-key AQCl7Kda5665NRAAUBO30oCCHiSHrnJUYSDQiQ== stdout: creating /var/lib/ceph/osd/ceph-7/keyring stdout: added entity osd.7 auth auth(auid = 18446744073709551615 key=AQCl7Kda5665NRAAUBO30oCCHiSHrnJUYSDQiQ== with 0 caps) Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-7/keyring Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-7/ Running command: chown -R ceph:ceph /dev/nvme1n1p2 Running command: ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 7 --monmap /var/lib/ceph/osd/ceph-7/activate.monmap --keyfile - --bluestore-block-db-path /dev/nvme1n1p2 --osd-data /var/lib/ceph/osd/ceph-7/ --osd-uuid e2b78628-53f5-4a78-870c-990d2063305a --setuser ceph --setgroup ceph --> ceph-volume lvm prepare successful for: /dev/mapper/bay7 Running command: ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-22c44762-565e-46b8-89aa-d379b6d42b2c/osd-block-e2b78628-53f5-4a78-870c-990d2063305a --path /var/lib/ceph/osd/ceph-7 Running command: ln -snf /dev/ceph-22c44762-565e-46b8-89aa-d379b6d42b2c/osd-block-e2b78628-53f5-4a78-870c-990d2063305a /var/lib/ceph/osd/ceph-7/block Running command: chown -R ceph:ceph /dev/dm-18 Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 Running command: ln -snf /dev/nvme1n1p2 /var/lib/ceph/osd/ceph-7/block.db Running command: chown -R ceph:ceph /dev/nvme1n1p2 Running command: systemctl enable ceph-volume@lvm-7-e2b78628-53f5-4a78-870c-990d2063305a stderr: Created symlink from /firstname.lastname@example.org to /usr/lib/systemd/system/ceph-volume@.service. Running command: systemctl start ceph-osd@7 --> ceph-volume lvm activate successful for osd ID: 7 --> ceph-volume lvm activate successful for osd ID: 7 --> ceph-volume lvm create successful for: /dev/mapper/bay7
#1 Updated by Alfredo Deza 11 months ago
- Status changed from New to In Progress
- Assignee set to Alfredo Deza
There is no support for multipath devices by ceph-volume for the use case you are presenting. The idea here is to allow someone who doesn't care about LVM to take a whole device and make it
a single logical volume group that has a single logical volume.
It is a naive/easy approach to avoid dealing with LVM.
If you want to use multipath then you must provide ceph-volume with a logical volume created from that multipath. We don't test multipath specifically, but we should not
have any problems working with a logical volume coming from a multipath device.
It is not as easy as just allowing 'mpath' when parsing TYPE, as I explain in :
Depending on the type of the multipath setup, if using an active/passive array as the underlying physical devices, filters are required in lvm.conf to exclude the disks that are part of those underlying devices. It is unfeasible for ceph-volume to understand what type of configuration is needed for LVM to be able to work in various different multipath scenarios. The functionality to create the LV for you is merely a (naive) convenience, anything that involves different settings or configuration must be provided by a config management system which can then provide VGs and LVs for ceph-volume to consume.
I am going to leave this open to address the documentation portion of it, because it should be clear that we will not support multipath to create a logical volume.
#3 Updated by Alfredo Deza 11 months ago
- Status changed from In Progress to Closed
As explained here, on the pull request for multipath support, and now on the ceph docs, we aren't going to allow working directly with a "raw" multipath device to create a logical volume.
If a logical volume is on top of a multipath device we should have no issues whatsoever (given correct configuration for LVM).
PR that documents multipath support https://github.com/ceph/ceph/pull/20878
merged commit 1cc5ad0 into master