image name metavariable
- When a VM has two RBD volumes attached, the admin socket only access one of them.
- Each RBD volume has its own admin socket but their name conflict and only one of them remains.
- No workaround.
- The new meta variable $cctid is defined and the conflicting
[client] admin socket = /var/run/ceph/rbd-$pid.asok
can be replaced by
[client] admin socket = /var/run/ceph/rbd-$pid-$cctid.asok
From Mike Dawson:
There are currently a few metavariables available for use in ceph.conf: http://ceph.com/docs/master/rados/configuration/ceph-conf/#metavariables In addition to those listed in that document, $pid was added in Bobtail. That allows me to get RBD admin sockets for libvirt/qemu guests by specifying [client] admin socket = /var/run/ceph/rbd-$pid.asok in /etc/ceph/ceph.conf. That works well for VMs with a single volume, but it does not work for VMs with multiple volumes. With multiple volumes, the admin only lists data on the last volume specified, leaving me unable to monitor stats on any volumes other than the last. I imagine you could specify a unique admin socket for each volume on the command-line that libvirt uses to spawn qemu, but that seems like a pain. Is it possible instead to add a metavariable that would expand to the image name? This way, my ceph.conf would look like [client] admin socket = /var/run/ceph/$imagename.asok and I would end up with admin sockets like: /var/run/ceph/rbd-volume-0025b693-6230-443e-ad6a-3aa43efdd648.asok /var/run/ceph/rbd-volume-009228f3-b8d7-4f3c-8458-0cdbd4809dc6.asok # rbd -p volumes ls volume-0025b693-6230-443e-ad6a-3aa43efdd648 volume-009228f3-b8d7-4f3c-8458-0cdbd4809dc6 I'd be happy to provide a patch if someone could give me sufficient guidance on how/where to add this functionality.
#1 Updated by Adam Crume about 9 years ago
The code would be in src/common.cc. I don't think it would be as simple as just adding another variable, because all of the existing variables are one-per-process, and a process can have multiple images open. The metavariable code would have to be made context-aware.
#2 Updated by Sage Weil about 9 years ago
It's per-cct. We can probably add some methods like
that would populate a map<string,string> and rerun the function that does the substitutions. I think part of the challenge will be making that happen early in the librbd setup, before the admin socket file is set up (probably before common_init() is called?). Should be doable, I think...
#3 Updated by Sage Weil about 9 years ago
...which is hard because librbd layers on top of an existing librados ioctx, which has already long been initialized. We can't even introduce our own default values for anything.
Perhaps the thing to do is make the asok init fail/skip creating the file is there is a metavariable still present (i.e., $ appears in the string) and make the librbd explicit fill it in and then re-trigger something on the CephContext? Something like reopen_logs(), but slightly more general to also include the asok file.
#5 Updated by Adam Crume about 9 years ago
This assumes that each image has its own cct, but a process could have multiple images open in one cct. (In fact, copy2 requires two open images.) Would we say that we don't support the image metavariable in that case?
What if an image is never opened? You'd never have a .asok.
Should the snap name be included? If so, should the metavariable have a different name?
I'd also like to point out that qemu opens and closes images twice, for some reason, just to make things more complicated.
#6 Updated by Adam Crume about 9 years ago
It's not perfect, but we could add a cctid variable so users could specify something like "admin socket = /var/run/ceph/rbd-$pid-$cctid.asok". This ensures at least that the ccts don't stomp on each others' files, although you'd still have to manually figure out which .asok is for which image.