ceph-disk: support indirect dmcrypt key retrieval
See Feb 2016 CDM session at:
See SUSE ftp implementation at:
ceph-disk: implement lockbox key management
Instead of storing the dmcrypt keys in the /etc/ceph/dmcrypt-keys
directory, they are stored in the monitor. If a machine with
OSDs created with ceph-disk prepare --dmcrypt is lost, it does
not contain the key that would allow to decrypt their content.
The dmcrypt key is retrieved from the monitor using a different keyring
for each OSD. It is stored in a small partition called the lockbox. At
boot time the lockbox is mounted
and used when the $uuid partition is detected by udev to map it with
The OSDs that were prepared prior to the lockbox implementation are
supported by looking up the key found in /etc/ceph/dmcrypt-keys before
looking in /var/lib/ceph/osd-lockbox/$uuid.
Signed-off-by: Loic Dachary <firstname.lastname@example.org>
#2 Updated by Loic Dachary over 3 years ago
osd dm-crypt key management scheme
- create new partition type OSD_LOCKBOX (or similar)
- populate with tiny file system
- automount, read-only, at /var/lib/ceph/osd-lockbox/$uuid (where uuid can be random, doesn't matter.. either unique to this device, or the osd uuid)
- 'km-mode' file indicates which key management scheme we are using.
- create lockbox partition on device (unencrypted), with tiny fs
- store luks key on monitor
- ceph config-key put dm-crypt/osd/$osd_uuid/luks $secret
- create a ceph user that can fetch it
- ceph auth get-or-create client.osd-lockbox.$osd_uuid mon 'allow command "config-key get" with key="dm-crypt/osd/$osd_uuid"' > /var/lib/ceph/osd-lockbox/$osd_uuid/keyring
- echo 'ceph-mon v1' > /var/lib/ceph/osd-lockbox/$osd_uuid/km-mode
- if km-mode == "ceph-mon v1" ...
- use user and key from 'keyring' file:
- 'ceph -n client.osd-lockbox.$osd_uuid -k /var/lib/ceph/osd-lockbox/$osd_uuid/keyring config-key get dm-crypt/osd/$osd_uuid' will write the secret to stdout
- when we encounter a dm-crypt device,
- first check legacy location (/etc/ceph/dmcrypt-keys/$osd_uuid)
- then check for lockbox (/var/lib/ceph/osd-lockbox/$osd_uuid)
- after we mount the lockbox, re-probe any dm-crypt devices with the same uuid (in case they tried before but lockbox wasn't mounted yet)
- if there are alternative key managers in use, indicate them in the lockbox, and adjust the "get key" method accordingly
#4 Updated by Loic Dachary over 3 years ago
[root@target167114242062 ceph-disk]# cryptsetup remove /dev/mapper/b2ba96fe-e05e-49a0-818b-9ba459ad6256 [root@target167114242062 ceph-disk]# cryptsetup --key-file /etc/ceph/dmcrypt-keys/b2ba96fe-e05e-49a0-818b-9ba459ad6256 create b2ba96fe-e05e-49a0-818b-9ba459ad6256 /dev/vdb1 --key-size 256 [root@target167114242062 ceph-disk]# mount -t xfs -o noatime,inode64 -- /dev/mapper/b2ba96fe-e05e-49a0-818b-9ba459ad6256 /mnt [root@target167114242062 ceph-disk]# umount /mnt [root@target167114242062 ceph-disk]# cryptsetup remove /dev/mapper/b2ba96fe-e05e-49a0-818b-9ba459ad6256 [root@target167114242062 ceph-disk]# cat /etc/ceph/dmcrypt-keys/b2ba96fe-e05e-49a0-818b-9ba459ad6256 | cryptsetup --key-file - create b2ba96fe-e05e-49a0-818b-9ba459ad6256 /dev/vdb1 --key-size 256 [root@target167114242062 ceph-disk]# mount -t xfs -o noatime,inode64 -- /dev/mapper/b2ba96fe-e05e-49a0-818b-9ba459ad6256 /mnt mount: wrong fs type, bad option, bad superblock on /dev/mapper/b2ba96fe-e05e-49a0-818b-9ba459ad6256, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so.