Ceph

BlueStore OSDs deployed with dmcrypt run fine until the host is rebooted, at which point udev fails to map the block partitions because it can't obtain the dmcrypt keys from a mon. The same behavior can be observed by stopping an OSD and closing the dmcrypt volume for its block partition and then attempting to activate it with ceph-disk.

I attempted to activate an OSD manually using 'ceph-disk activate-block --dmcrypt <block dev>' and got an error (using -v) about the OSD not having a cluster ID. When I searched for the error in the ceph-disk source, I discovered that the issue was a missing ceph_fsid file containing the cluster fsid in the OSD's lockbox, which is required in order for ceph-disk to query a mon for the dmcrypt key for the block partition.

Once I wrote the cluster fsid to a new ceph_fsid file in all of my OSD lockbox filesystems, all of my OSDs were able to start using 'ceph-disk activate-block' and were also able to start automatically when the host was rebooted.