Ceph » rbd

The RBD image directory is stored within an object named 'rbd_directory' in each pool. You could create a capspec which allows one to R/W RBD images but not read/write the 'rbd_directory' object. This would effectively prevent you from creating/deleting images using the client with this restricted capspec, so I am not sure if this is valid with your use-case (i.e. this might be valid for protecting Nova Compute nodes attaching RBD cinder volumes, but not Cinder nodes nor Nova Compute nodes using RBD for ephemeral storage).

ceph auth caps <client name> mon 'allow r' osd 'allow x object_prefix rbd_children, allow rwx object_prefix rbd_header., allow rwx object_prefix rbd_id., allow rw object_prefix rbd_data.'

Let me try this and see if it will do what we think. I don't know enough about the Open Stack side, but I hope we can use one key for Cinder/Glance and a different key for Nova.

I've been trying to find where all these CAPs are documented. Can you point me in the right direction?

Yes, you should be able to use different users within Nova, Cinder, and Glance config files. The capability grammar is documented here: https://ceph.com/docs/v0.79/man/8/ceph-authtool/. Since there is no RBD server (the RBD client just manipulates objects within RADOS to present a block storage device), there isn't a high-level set of capabilities for high-level RBD actions.

I've looked over that document a few times, but I'm not finding specifics about things like "object_prefix", "rbd_header", etc. I'm not sure what is available and what they do to understand what is needed. Thanks.

Using those caps does not allow the kernel client to mount the image:

[root@nodezz ~]# ceph auth caps client.rdleblanc mon 'allow r' osd 'allow x object_prefix rbd_children, allow rwx object_prefix rbd_header., allow rwx object_prefix rbd_id., allow rw object_prefix rbd_data.'
updated caps for client.rdleblanc

[root@nodezz ~]# rbd --user rdleblanc -p test map junk
rbd: add failed: (1) Operation not permitted

If I relax the caps (to make sure it would work with the keyring, etc), I can mount it. I tried with and without the periods in the ceph auth caps line.

I apologize, I thought I mentioned that you need to use RBD image format 2, but re-reading my comments I seemed to have left that part off. OpenStack will automatically create format 2 RBD images (and it uses the user-mode librbd library instead of the kernel module).

# ceph auth caps client.rbd mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow class-write object_prefix rbd_directory, allow rwx object_prefix rbd_header., allow rwx object_prefix rbd_data., allow rwx object_prefix rbd_id.'
# rbd create --size 10 test
# rbd ls -l
NAME   SIZE PARENT FMT PROT LOCK 
test 10240k          2    
# sudo rbd -n client.rbd map test
/dev/rbd0

Yes, working with Format 2 images, it indeed does mount without allowing listing. I've confirmed that our OpenStack environment is using version 2. For completeness, I've included my tests.

[root@nodezz test12]# rbd -p test ls -l
NAME SIZE PARENT FMT PROT LOCK
perms_test2 1024M 1
test1 102400M 1
test10 102400M 1
test11 1024G 1
test12 1024G 1
test2 102400M 1
test3 102400M 1
test4 102400M 1
test5 102400M 1
test6 102400M 1
test7 102400M 1
test8 102400M 1
test9 102400M 1
perms_test 10240k 2
[root@nodezz test12]# rbd -p test -n client.rdleblanc ls -l
rbd: list: (1) Operation not permitted
[root@nodezz test12]# rbd -p test -n client.rdleblanc map perms_test2
rbd: add failed: (1) Operation not permitted
[root@nodezz test12]# rbd -p test -n client.rdleblanc map perms_test
[root@nodezz test12]# ls /dev/rbd/test/
perms_test test10 test12 test5 test6 test7 test8
[root@nodezz test12]# rbd -p test -n client.rdleblanc unmap /dev/rbd/test/perms_test
[root@nodezz test12]# ceph auth get client.rdleblanc
exported keyring for client.rdleblanc
[client.rdleblanc]
key = ASECRETKEY==
caps mon = "allow r"
caps osd = "allow x object_prefix rbd_children, allow rwx object_prefix rbd_header., allow rwx object_prefix rbd_id., allow rw object_prefix rbd_data."

This satisfies what we were trying to accomplish. Is there some documentation on the available permissions (object_prefix, rbd_directory, rbd_header, etc...)? Please close the ticket. Thank you.

OK, so one more question. This looks like it allows access to any pool. Is there a way to limit this to a particular pool? I've tried adding pool=test after each cap, but I get "rbd: add failed: (34) Numerical result out of range"?

Try placing the "pool=test" argument before the "object_prefix XYZ" portion of the cap:

# ceph auth caps client.rbd mon 'allow r' osd 'allow x pool=test object_prefix rbd_children, allow rwx pool=test object_prefix rbd_header., allow rwx pool=test object_prefix rbd_id., allow rw pool=test object_prefix rbd_data.'

OK, putting the pool argument first does work. We have consequently found out that Nova does require list permissions because it tries to import the image.

2014-10-21 13:23:24.639 19220 TRACE nova.compute.utils [instance: c766c0cb-139f-4cb0-b4d0-16604d3a4dd5] Command: rbd import --pool vms /var/lib/nova/instances/_base/514b5558848890756b9f6e319c4bd7818c1d8739 c766c0cb-139f-4cb0-b4d0-16604d3a4dd5_disk --new-format --id nova --conf /etc/ceph/ceph.conf
2014-10-21 13:23:24.639 19220 TRACE nova.compute.utils [instance: c766c0cb-139f-4cb0-b4d0-16604d3a4dd5] Exit code: 1
2014-10-21 13:23:24.639 19220 TRACE nova.compute.utils [instance: c766c0cb-139f-4cb0-b4d0-16604d3a4dd5] Stdout: ''
2014-10-21 13:23:24.639 19220 TRACE nova.compute.utils [instance: c766c0cb-139f-4cb0-b4d0-16604d3a4dd5] Stderr: 'rbd: image creation failed\n\rImporting image: 0% complete...failed.\nrbd: import failed: (1) Operation not permitted\n2014-10-21 13:23:24.097711 7fb68a1d3760 -1 librbd: Could not tell if c766c0cb-139f-4cb0-b4d0-16604d3a4dd5_disk already exists\n'


All of our images are being stored in Ceph so it should be just creating a snapshot, but I think we will still run into the same issue if Nova is the one trying to create the snapshot. I'll keep digging into this. Thanks for all your help so far.

It sounds like Nova is configured to use RBD as the backing store for its ephemeral disk images instead of the local filesystem. In this case, the provided caps cannot be used because they will prevent Nova from creating images in the RBD pool. To support your use-case, the caps syntax would need to be expanded to restrict access to specific OSD class methods.

Is the list of OSD class methods documented somewhere?