Ceph » ceph-volume

Environment: Quincy 17.2.5

I am trying to create a Ceph cluster through cephadm using multipath devices. According to the latest documentation [1], ceph-volume should be able to lvm prepare these devices if LVM is configured with multipath_component_detection = 1 (which is the case on my system).

Indeed, `ceph orch device ls` shows only the multipath devices, there are no duplicates like in `lsblk`:

[ceph: root@ceph000 /]# ceph orch device ls 
HOST                 PATH                           TYPE  DEVICE ID                                   SIZE  AVAILABLE  REFRESHED  REJECT REASONS  
host000.example.com  /dev/mapper/35000c500d9ac263b  hdd                                              18.0T  Yes        21m ago                    
host000.example.com  /dev/mapper/35000c500d9db3283  hdd                                              18.0T  Yes        21m ago                   
[continued...]

However, applying an OSD spec to use all these devices results in: [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.osd-hdd

/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b3eade34-7e5e-4d53-aa15-3baa43fe564e                                                                                         
/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes                                                                                                                                                                                                            
/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it                                                                                                
/bin/docker: stderr  stderr: purged osd.0                                                                                                    
/bin/docker: stderr Traceback (most recent call last):                                                                                                                                                                                                                                     
/bin/docker: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare                                                                                                                                                                
/bin/docker: stderr     self.prepare()                                                                                                                                                                                                                                                     
/bin/docker: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root                                                                                                                                                                               
/bin/docker: stderr     return func(*a, **kw)                                                                                                                                                                                                                                              
/bin/docker: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 363, in prepare                                                                                                                                                                     
/bin/docker: stderr     block_lv = self.prepare_data_device('block', osd_fsid)                                                                                                                                                                                                             
/bin/docker: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 226, in prepare_data_device                                                                                                                                                         
/bin/docker: stderr     raise RuntimeError(' '.join(error))                                                                                  
/bin/docker: stderr RuntimeError: Cannot use device (/dev/mapper/35000c500d9ac263b). A vg/lv path or an existing device is needed

I have traced the error to the following code:

        if disk.is_partition(device) or disk.is_device(device):
[...]
        else:
            error = [
                'Cannot use device ({}).'.format(device),
                'A vg/lv path or an existing device is needed']
            raise RuntimeError(' '.join(error))

I am guessing disk.is_device returned False which caused the execution of the else statement. Definition of disk.is_device():

def is_device(dev):
    """ 
    Boolean to determine if a given device is a block device (**not**
    a partition!) 

    For example: /dev/sda would return True, but not /dev/sdc1
    """ 
    if not os.path.exists(dev):
        return False
    if not dev.startswith('/dev/'):
        return False
    if dev[len('/dev/'):].startswith('loop'):
        if not allow_loop_devices():
            return False

    # fallback to stat
    return _stat_is_device(os.lstat(dev).st_mode)

We would reach the "fallback to stat" code, but it calls os.lstat() which does not follow symbolic links. However our device is a symbolic link, so it will return False:

$ ls -l /dev/mapper/35000c500d9ac263b
lrwxrwxrwx 1 root root 8 Apr  7 10:03 /dev/mapper/35000c500d9ac263b -> ../dm-22

I changed the os.lstat() call to os.stat instead and it resolves the issue, but I don't know if it could have unintended effects in other scenarios.

[1] https://docs.ceph.com/en/latest/ceph-volume/lvm/prepare/#multipath-support