Feature #11881
Updated by Loïc Dachary over 8 years ago
h3. Need review * ceph : https://github.com/ceph/ceph/pull/5699 https://github.com/ceph/ceph/pull/5584 * ceph-qa-suite : https://github.com/ceph/ceph-qa-suite/pull/546 * teuthology : https://github.com/ceph/teuthology/pull/606 h3. What needs to be fixed in ceph-disk We have these udev rules * https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules that call ceph-disk when they see GPT labels indicating the partition is used for ceph OSD data or journal. When connecting a device with multipath, what probably happens is that the original device shows up twice (for both paths), and then there is a dm device that we should be consuming. Instead, we're triggering ceph-disk on both of the underlying devices (and not on the dm one). We can make ceph-disk ignore a device if it appears to be a slave (i.e., it appears as /sys/block/*/slaves/$DEV for some other device). This may race if the dm setup is also udev driven. h3. What other udev mechanism we need to use ? * DM_MULTIPATH_DEVICE_PATH udev environment variable. This is set by 62-multipath.rules for all devices that will become paths in a multipath device, and all their partitions. There are a couple gotchas, however. ** It won't get set the very first time a piece of storage hardware is seen by the system. With find_multipaths enabled, there's no way for multipath to know ahead of time if a new piece of storage will be multipathable when it first appears. ** If the storage gets discovered in the initramfs, multipath won't be able to update it's list of multipathable devices there. This means it will fail to get correctly labelled in the initramfs every time, until the initramfs is remade. That's why it is advisable to remake the initramfs after adding new hardware to a multipath system (there are ways of working around this if that's incredibly important). But aside from new hardware (which probably won't be labelled for ceph-disk the first time it appears anyway), this should work to keep you from messing with multipath's path devices. h3. Set up multipath without actually having any multipath hardware * modprobe scsi_debug vpd_use_hostno=0 add_host=2 dev_size_mb=100 will get you 2 100MB scsi_debug devices that multipath will happily set itself up on. But I don't know if this will help you, since you can't choose what's on the device when it gets discovered. The device usually (always?) comes up with the same WWID, so after the first time, the multipath udev rules will flag it correctly, but I doubt that the 95-ceph-osd.rules would pick it up. h3. With an unused scsi device * multipath <scsi_device> Multipath will make a multipath device with only one path on top of it, and will remember that it is supposed to be multipathed in the future. Once you're all done with it, run: * multipath -f <multipath_device> to remove the multipath device, and * multipath -w <scsi_device> To wipe the information that it's supposed to be a multipath device (so that it won't automatically get multipathed the next time you start up). If the unused scsi is a normal disk (say /dev/sdb) this allows testing everything *except* that another device (say, /dev/sdf) won't appear that has the same disk. h3. Testing with multiple devices Setup an iscsi session, duplicate it by * Getting the sessionid: iscsiadm -m session -P1 | grep SID * Create a new session with the same data: iscsiadm -m session -r <sessionid> -o new This should create another iscsi device for each of the existing session, and multipathd will notice that there are two paths to the same disk, and automatically create a multipath device on it. h3. Additional information See also https://github.com/ceph/ceph/pull/1229