Feature #11881
closedceph-disk support for multipath
0%
Description
Need review¶
- ceph : https://github.com/ceph/ceph/pull/5699
- ceph-qa-suite : https://github.com/ceph/ceph-qa-suite/pull/546
- teuthology : https://github.com/ceph/teuthology/pull/606
What needs to be fixed in ceph-disk¶
We have these udev rules
that call ceph-disk when they see GPT labels indicating the partition is
used for ceph OSD data or journal. When connecting a device with multipath, what probably happens is that the original device shows up twice (for both paths), and then there is a dm device that we should be consuming. Instead, we're triggering ceph-disk on both of the underlying devices (and not on the dm one).
We can make ceph-disk ignore a device if it appears to be a slave (i.e.,
it appears as /sys/block/*/slaves/$DEV for some other device). This may race if the dm setup is also udev driven.
What other udev mechanism we need to use ?¶
- DM_MULTIPATH_DEVICE_PATH udev environment variable. This is set by 62-multipath.rules for all devices that will become paths in a multipath device, and all their partitions. There are a couple gotchas, however.
- It won't get set the very first time a piece of storage hardware is seen by the system. With find_multipaths enabled, there's no way for multipath to know ahead of time if a new piece of storage will be multipathable when it first appears.
- If the storage gets discovered in the initramfs, multipath won't be able to update it's list of multipathable devices there. This means it will fail to get correctly labelled in the initramfs every time, until the initramfs is remade. That's why it is advisable to remake the initramfs after adding new hardware to a multipath system (there are ways of working around this if that's incredibly important).
But aside from new hardware (which probably won't be labelled for
ceph-disk the first time it appears anyway), this should work to keep
you from messing with multipath's path devices.
Set up multipath without actually having any multipath hardware¶
- modprobe scsi_debug vpd_use_hostno=0 add_host=2 dev_size_mb=100
will get you 2 100MB scsi_debug devices that multipath will happily set
itself up on. But I don't know if this will help you, since you can't
choose what's on the device when it gets discovered. The device usually
(always?) comes up with the same WWID, so after the first time, the
multipath udev rules will flag it correctly, but I doubt that the
95-ceph-osd.rules would pick it up.
With an unused scsi device¶
- multipath <scsi_device>
Multipath will make a multipath device with only one path on top of it,
and will remember that it is supposed to be multipathed in the future.
Once you're all done with it, run:
- multipath -f <multipath_device>
to remove the multipath device, and
- multipath -w <scsi_device>
To wipe the information that it's supposed to be a multipath device (so
that it won't automatically get multipathed the next time you start up).
If the unused scsi is a normal disk (say /dev/sdb) this allows testing everything except that another device (say, /dev/sdf) won't appear that has the same disk.
Testing with multiple devices¶
Setup an iscsi session, duplicate it by
- Getting the sessionid: iscsiadm -m session -P1 | grep SID
- Create a new session with the same data: iscsiadm -m session -r <sessionid> -o new
This should create another iscsi device for each of the existing
session, and multipathd will notice that there are two paths to the
same disk, and automatically create a multipath device on it.
Additional information¶
Files