Ceph

Did you perhaps add the cephx option after enabling your monitors? If you tell the client to use cephx but it's not enabled in the monitors things aren't going to work.

I just tested again, ensuring a complete ceph restart on all devices after setting auth cephx. The mapping failed as above.

Can you post the output of 'strace -f rbd map test1'?

What kernel version are you using? That error is coming directly from the kernel rbd module.

self-compiled 3.4.12

After you the map fails, is there anything added to dmesg?

Augh. Sorry, I should have noticed that before:

[79683.055935] libceph: client0 fsid c6618970-0ce0-4cb2-bc9a-dd5f29b62e24
[79683.059286] libceph: error -2 building auth method x request

The only path I'm seeing that could lead to that error is if the kernel wasn't compiled with support for "cbc(aes)" with the CRYPTO_ALG_ASYNC flag. I'd guess any of the CRYTPO_AES* build options (with CRYPTO_BLKCIPHER2?) would be sufficient.

Aha! That did it, thanks.

After installing cbc.ko (already had aes):

# grep auth /etc/ceph/ceph.conf 
      auth supported = cephx
# modprobe rbd
# lsmod | egrep 'aes|cbc|rbd'
rbd                    19378  0 
libceph               122328  1 rbd
# rbd map test1
# rbd showmapped
id    pool    image    snap    device
1    rbd    test1    -    /dev/rbd1
# lsmod | egrep 'aes|cbc|rbd'
aes_x86_64              7920  0 
aes_generic            27295  1 aes_x86_64
cbc                     2814  0 
rbd                    19378  1 
libceph               122328  1 rbd

Interesting, no google hits on the 2nd "error -2 building auth" message. Perhaps the kernel message and/or docs could be enhanced to suggest a solution? (Not that it would have helped me until you prompted me to check!). Of course, if/when Google indexes this page the point becomes moot!