Ceph

After reboot, my OSDS are not started.
I have one harddisk per OSD, with the journal on a partition on /dev/sda, which is an ssd.
/dev/sda also contains an LVM volume group for the root fs. OS is CentOS 7.2. Ceph version 10.1.2.

I enabled debug logging in udev, and found this (rest of the output in the attached log):

Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶⁴⁵: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sda4' [812] exit with return code 1
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶³³: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sda11'(err) ' load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()'
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶⁵¹: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sda9'(err) 'Traceback (most recent call last):'
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶⁵⁴: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdk1'(err) ' load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()'
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶²⁸: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdj1'(err) ' load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()'
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶⁴⁷: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sda6'(err) ' File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4906, in main'
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶²⁷: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdf1'(err) ' setup_statedir(args.statedir)'
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶³⁶: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdl1'(err) ' setup_statedir(args.statedir)'
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶²³: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdb1'(err) 'Traceback (most recent call last):'
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶²⁹: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdh1'(err) ' os.mkdir(STATEDIR)'
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶²⁵: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sde1'(err) ' setup_statedir(args.statedir)'
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶³²: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sda10'(err) ' File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4366, in setup_statedir'
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶⁴⁶: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sda5'(err) 'OSError: [Errno 2] No such file or directory: '/var/lib/ceph''
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶⁴¹: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sda14'(err) 'OSError: [Errno 2] No such file or directory: '/var/lib/ceph''
Apr 20 20:53:06 ams1-pod11-ceph18.tilaa.nl systemd-udevd⁶³⁹: '/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sda13'(err) 'OSError: [Errno 2] No such file or directory: '/var/lib/ceph''

I'll attach the full debug log.

It's hard to tell from the enormous amount of output, but what I think is happening is that:
- udev runs from initramfs
- at this point lvm isn't done scanning for vgs / pvs and the root fs is not available yet
- ceph disk tries to create /var/lib/ceph on the root filesystem.

I can't trigger this on every reboot however, I guess it depends on how fast LVM assembles its devices.

As a sidenote, what makes debugging this hard is that those python tracebacks only show up with udev in debug mode.