Bug #59375
openceph-volume should support symbolic links to devices e.g. for multipath
0%
Description
Environment: Quincy 17.2.5
I am trying to create a Ceph cluster through cephadm using multipath devices. According to the latest documentation [1], ceph-volume should be able to lvm prepare these devices if LVM is configured with multipath_component_detection = 1 (which is the case on my system).
Indeed, `ceph orch device ls` shows only the multipath devices, there are no duplicates like in `lsblk`:
[ceph: root@ceph000 /]# ceph orch device ls HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS host000.example.com /dev/mapper/35000c500d9ac263b hdd 18.0T Yes 21m ago host000.example.com /dev/mapper/35000c500d9db3283 hdd 18.0T Yes 21m ago [continued...]
However, applying an OSD spec to use all these devices results in: [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.osd-hdd
/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b3eade34-7e5e-4d53-aa15-3baa43fe564e /bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes /bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it /bin/docker: stderr stderr: purged osd.0 /bin/docker: stderr Traceback (most recent call last): /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare /bin/docker: stderr self.prepare() /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /bin/docker: stderr return func(*a, **kw) /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 363, in prepare /bin/docker: stderr block_lv = self.prepare_data_device('block', osd_fsid) /bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 226, in prepare_data_device /bin/docker: stderr raise RuntimeError(' '.join(error)) /bin/docker: stderr RuntimeError: Cannot use device (/dev/mapper/35000c500d9ac263b). A vg/lv path or an existing device is needed
I have traced the error to the following code:
if disk.is_partition(device) or disk.is_device(device): [...] else: error = [ 'Cannot use device ({}).'.format(device), 'A vg/lv path or an existing device is needed'] raise RuntimeError(' '.join(error))
I am guessing disk.is_device returned False which caused the execution of the else statement. Definition of disk.is_device():
def is_device(dev): """ Boolean to determine if a given device is a block device (**not** a partition!) For example: /dev/sda would return True, but not /dev/sdc1 """ if not os.path.exists(dev): return False if not dev.startswith('/dev/'): return False if dev[len('/dev/'):].startswith('loop'): if not allow_loop_devices(): return False # fallback to stat return _stat_is_device(os.lstat(dev).st_mode)
We would reach the "fallback to stat" code, but it calls os.lstat() which does not follow symbolic links. However our device is a symbolic link, so it will return False:
$ ls -l /dev/mapper/35000c500d9ac263b lrwxrwxrwx 1 root root 8 Apr 7 10:03 /dev/mapper/35000c500d9ac263b -> ../dm-22
I changed the os.lstat() call to os.stat instead and it resolves the issue, but I don't know if it could have unintended effects in other scenarios.
[1] https://docs.ceph.com/en/latest/ceph-volume/lvm/prepare/#multipath-support
Updated by Pierre Riteau about 1 year ago
Pull request: https://github.com/ceph/ceph/pull/50938
Updated by Guillaume Abrioux about 1 year ago
- Status changed from New to Fix Under Review
- Assignee set to Pierre Riteau
- Backport set to reef,quincy,pacific
- Pull request ID set to 50938
Updated by Pierre Riteau about 2 months ago
This can be closed now, it was fixed by https://github.com/ceph/ceph/pull/53309.
Updated by Konstantin Shalygin 26 days ago
- Status changed from Fix Under Review to Pending Backport
- Backport changed from reef,quincy,pacific to reef quincy squid
- Pull request ID changed from 50938 to 53309