Separate rbd listing into CAP
We are concerned that if the key is compromised in our OpenStack environment, then all images in the pool can be listed, exported, etc. Since OpenStack knows which images are in the Ceph pool, it should be able to request a mount of the appropriate image by name without needing the ability to list. By making rbd listing a cap that can be enabled/disabled, we can feel better that if the key were to be compromised it would add a first layer of defense. A person would not be able to easily list images and would have to guess at UUIDs to try operations on. I'm sure that the person could find the UUIDs of the VMs running on the local compromised host through ps/proc, but at least it would limit them to a much smaller percentage.
It would be similar to having only execute permissions on a directory in Linux. If you have rights to a file/folder underneath and know the name, you can operate on the file/folder.
#1 Updated by Jason Dillaman over 4 years ago
The RBD image directory is stored within an object named 'rbd_directory' in each pool. You could create a capspec which allows one to R/W RBD images but not read/write the 'rbd_directory' object. This would effectively prevent you from creating/deleting images using the client with this restricted capspec, so I am not sure if this is valid with your use-case (i.e. this might be valid for protecting Nova Compute nodes attaching RBD cinder volumes, but not Cinder nodes nor Nova Compute nodes using RBD for ephemeral storage).
ceph auth caps <client name> mon 'allow r' osd 'allow x object_prefix rbd_children, allow rwx object_prefix rbd_header., allow rwx object_prefix rbd_id., allow rw object_prefix rbd_data.'
#2 Updated by Robert LeBlanc over 4 years ago
Let me try this and see if it will do what we think. I don't know enough about the Open Stack side, but I hope we can use one key for Cinder/Glance and a different key for Nova.
I've been trying to find where all these CAPs are documented. Can you point me in the right direction?
#3 Updated by Jason Dillaman over 4 years ago
Yes, you should be able to use different users within Nova, Cinder, and Glance config files. The capability grammar is documented here: https://ceph.com/docs/v0.79/man/8/ceph-authtool/. Since there is no RBD server (the RBD client just manipulates objects within RADOS to present a block storage device), there isn't a high-level set of capabilities for high-level RBD actions.
#5 Updated by Robert LeBlanc over 4 years ago
Using those caps does not allow the kernel client to mount the image:
[root@nodezz ~]# ceph auth caps client.rdleblanc mon 'allow r' osd 'allow x object_prefix rbd_children, allow rwx object_prefix rbd_header., allow rwx object_prefix rbd_id., allow rw object_prefix rbd_data.'
updated caps for client.rdleblanc
[root@nodezz ~]# rbd --user rdleblanc -p test map junk
rbd: add failed: (1) Operation not permitted
If I relax the caps (to make sure it would work with the keyring, etc), I can mount it. I tried with and without the periods in the ceph auth caps line.
#6 Updated by Jason Dillaman over 4 years ago
I apologize, I thought I mentioned that you need to use RBD image format 2, but re-reading my comments I seemed to have left that part off. OpenStack will automatically create format 2 RBD images (and it uses the user-mode librbd library instead of the kernel module).
# ceph auth caps client.rbd mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow class-write object_prefix rbd_directory, allow rwx object_prefix rbd_header., allow rwx object_prefix rbd_data., allow rwx object_prefix rbd_id.' # rbd create --size 10 test # rbd ls -l NAME SIZE PARENT FMT PROT LOCK test 10240k 2 # sudo rbd -n client.rbd map test /dev/rbd0
#7 Updated by Robert LeBlanc over 4 years ago
Yes, working with Format 2 images, it indeed does mount without allowing listing. I've confirmed that our OpenStack environment is using version 2. For completeness, I've included my tests.
[root@nodezz test12]# rbd -p test ls -l
NAME SIZE PARENT FMT PROT LOCK
perms_test2 1024M 1
test1 102400M 1
test10 102400M 1
test11 1024G 1
test12 1024G 1
test2 102400M 1
test3 102400M 1
test4 102400M 1
test5 102400M 1
test6 102400M 1
test7 102400M 1
test8 102400M 1
test9 102400M 1
perms_test 10240k 2
[root@nodezz test12]# rbd -p test -n client.rdleblanc ls -l
rbd: list: (1) Operation not permitted
[root@nodezz test12]# rbd -p test -n client.rdleblanc map perms_test2
rbd: add failed: (1) Operation not permitted
[root@nodezz test12]# rbd -p test -n client.rdleblanc map perms_test
[root@nodezz test12]# ls /dev/rbd/test/
perms_test test10 test12 test5 test6 test7 test8
[root@nodezz test12]# rbd -p test -n client.rdleblanc unmap /dev/rbd/test/perms_test
[root@nodezz test12]# ceph auth get client.rdleblanc
exported keyring for client.rdleblanc
key = ASECRETKEY==
caps mon = "allow r"
caps osd = "allow x object_prefix rbd_children, allow rwx object_prefix rbd_header., allow rwx object_prefix rbd_id., allow rw object_prefix rbd_data."
This satisfies what we were trying to accomplish. Is there some documentation on the available permissions (object_prefix, rbd_directory, rbd_header, etc...)? Please close the ticket. Thank you.
#9 Updated by Jason Dillaman about 4 years ago
Try placing the "pool=test" argument before the "object_prefix XYZ" portion of the cap:
# ceph auth caps client.rbd mon 'allow r' osd 'allow x pool=test object_prefix rbd_children, allow rwx pool=test object_prefix rbd_header., allow rwx pool=test object_prefix rbd_id., allow rw pool=test object_prefix rbd_data.'
#10 Updated by Robert LeBlanc about 4 years ago
OK, putting the pool argument first does work. We have consequently found out that Nova does require list permissions because it tries to import the image.
2014-10-21 13:23:24.639 19220 TRACE nova.compute.utils [instance: c766c0cb-139f-4cb0-b4d0-16604d3a4dd5] Command: rbd import --pool vms /var/lib/nova/instances/_base/514b5558848890756b9f6e319c4bd7818c1d8739 c766c0cb-139f-4cb0-b4d0-16604d3a4dd5_disk --new-format --id nova --conf /etc/ceph/ceph.conf
2014-10-21 13:23:24.639 19220 TRACE nova.compute.utils [instance: c766c0cb-139f-4cb0-b4d0-16604d3a4dd5] Exit code: 1
2014-10-21 13:23:24.639 19220 TRACE nova.compute.utils [instance: c766c0cb-139f-4cb0-b4d0-16604d3a4dd5] Stdout: ''
2014-10-21 13:23:24.639 19220 TRACE nova.compute.utils [instance: c766c0cb-139f-4cb0-b4d0-16604d3a4dd5] Stderr: 'rbd: image creation failed\n\rImporting image: 0% complete...failed.\nrbd: import failed: (1) Operation not permitted\n2014-10-21 13:23:24.097711 7fb68a1d3760 -1 librbd: Could not tell if c766c0cb-139f-4cb0-b4d0-16604d3a4dd5_disk already exists\n'
All of our images are being stored in Ceph so it should be just creating a snapshot, but I think we will still run into the same issue if Nova is the one trying to create the snapshot. I'll keep digging into this. Thanks for all your help so far.
#11 Updated by Jason Dillaman about 4 years ago
It sounds like Nova is configured to use RBD as the backing store for its ephemeral disk images instead of the local filesystem. In this case, the provided caps cannot be used because they will prevent Nova from creating images in the RBD pool. To support your use-case, the caps syntax would need to be expanded to restrict access to specific OSD class methods.