Ceph

Reported by Sylvain Munaut ("tnt" on OFTC):

(12:30:27) tnt: Is mounting a RBD on a machine that has an OSD supposed to kernel panic ?
(12:33:20) fghaas: I'd say kernel panics are never supposed to happen
(12:33:49) fghaas: however are you sure it's a panic and not just a stack trace?
(12:33:49) tnt: http://pastebin.com/RgnapSvk
(12:33:56) tnt: Well the machine is dead
(12:34:34) tnt: and it's 100% reproductible on that setup ... (I rebooted it and retried a rbd map and bam ... dead again)
(12:34:41) fghaas: is this right after "rbd map" or after actually opening your /dev/rbdX?
(12:35:12) tnt: right after rbd map. I don't even see it returning, the ssh connection dies before that.
(12:35:47) fghaas: 0.48, ubuntu 3.2.0 kernel? does this also occur when you're mapping from a different box?
(12:37:38) tnt: It's 0.48 and a up to date ubuntu 12.04 ( so yes, 3.2.0-24-generic kernel ). I'll try from another box.
(12:42:10) tnt: fghaas: seems to work fine from another box
(12:42:36) fghaas: ok, that's what I thought. I'd be surprised if it didn't, it's been working fine for me that way
(12:42:45) fghaas: never tried mounting from an osd though
(12:43:15) fghaas: now it does say somewhere that you shouldn't be mounting cephfs from a ceph cluster nodes, but afair nothing about rbd
(12:43:38) fghaas: but the problem seems to be in libceph, which both use
(12:44:14) tnt: strangly I used the manual "echo ..." method and ... it worked fine.
(12:44:38) fghaas: you mean via the sysfs file?
(12:45:36) tnt: Yup ...
(12:45:40) fghaas: what's your exact "rbd map" command line?
(12:45:52) tnt: rbd map es-data-0 --pool rbd --name client.admin --secret /etc/ceph/keyring
(12:46:04) tnt: and for the echo I do:
(12:46:09) tnt: echo "10.208.3.67 name=admin,secret=AQDgARFQ8MzbLxAA5bouqkjb26sZk5eGqfkKcQ== rbd es-data-0" > /sys/bus/rbd/add
(12:47:07) fghaas: um, wait. it's not supposed to work with --secret pointing to the standard keyring file
(12:47:46) fghaas: do echo AQDgARFQ8MzbLxAA5bouqkjb26sZk5eGqfkKcQ== > /tmp/secret, and then rbd map es-data-0 --pool rbd --name client.admin --secret /tmp/secret
(12:48:09) tnt: Ah yes indeed, this way it works !