Feature #6228
closed
Added by Mark Nelson over 10 years ago.
Updated about 9 years ago.
Backport:
giant,firefly,dumpling
Description
User description¶
Symptoms:
- When a VM has two RBD volumes attached, the admin socket only access one of them.
Diagnostic:
- Each RBD volume has its own admin socket but their name conflict and only one of them remains.
Workaround:
Fix:
Description¶
From Mike Dawson:
There are currently a few metavariables available for use in ceph.conf:
http://ceph.com/docs/master/rados/configuration/ceph-conf/#metavariables
In addition to those listed in that document, $pid was added in Bobtail. That allows me to get RBD admin sockets for libvirt/qemu guests by specifying
[client]
admin socket = /var/run/ceph/rbd-$pid.asok
in /etc/ceph/ceph.conf. That works well for VMs with a single volume, but it does not work for VMs with multiple volumes. With multiple volumes, the admin only lists data on the last volume specified, leaving me unable to monitor stats on any volumes other than the last.
I imagine you could specify a unique admin socket for each volume on the command-line that libvirt uses to spawn qemu, but that seems like a pain.
Is it possible instead to add a metavariable that would expand to the image name? This way, my ceph.conf would look like
[client]
admin socket = /var/run/ceph/$imagename.asok
and I would end up with admin sockets like:
/var/run/ceph/rbd-volume-0025b693-6230-443e-ad6a-3aa43efdd648.asok
/var/run/ceph/rbd-volume-009228f3-b8d7-4f3c-8458-0cdbd4809dc6.asok
# rbd -p volumes ls
volume-0025b693-6230-443e-ad6a-3aa43efdd648
volume-009228f3-b8d7-4f3c-8458-0cdbd4809dc6
I'd be happy to provide a patch if someone could give me sufficient guidance on how/where to add this functionality.
The code would be in src/common.cc. I don't think it would be as simple as just adding another variable, because all of the existing variables are one-per-process, and a process can have multiple images open. The metavariable code would have to be made context-aware.
It's per-cct. We can probably add some methods like
cct->set_meta("image", "foo");
that would populate a map<string,string> and rerun the function that does the substitutions. I think part of the challenge will be making that happen early in the librbd setup, before the admin socket file is set up (probably before common_init() is called?). Should be doable, I think...
...which is hard because librbd layers on top of an existing librados ioctx, which has already long been initialized. We can't even introduce our own default values for anything.
Perhaps the thing to do is make the asok init fail/skip creating the file is there is a metavariable still present (i.e., $ appears in the string) and make the librbd explicit fill it in and then re-trigger something on the CephContext? Something like reopen_logs(), but slightly more general to also include the asok file.
- Status changed from New to In Progress
- Assignee set to Adam Crume
This assumes that each image has its own cct, but a process could have multiple images open in one cct. (In fact, copy2 requires two open images.) Would we say that we don't support the image metavariable in that case?
What if an image is never opened? You'd never have a .asok.
Should the snap name be included? If so, should the metavariable have a different name?
I'd also like to point out that qemu opens and closes images twice, for some reason, just to make things more complicated.
It's not perfect, but we could add a cctid variable so users could specify something like "admin socket = /var/run/ceph/rbd-$pid-$cctid.asok". This ensures at least that the ccts don't stomp on each others' files, although you'd still have to manually figure out which .asok is for which image.
Yeah, that is probably a good idea anyway.. we've had uniqueness issues like this before! That is an easy thing and solves most of the problem
- Status changed from In Progress to Fix Under Review
- Status changed from Fix Under Review to Pending Backport
- Backport set to giant,firefly
Glad to see this feature added. Thank you! Mark, Adam, Sage, and Loic!
Assuming it wouldn't be too difficult, could this be tagged to be backported to Dumpling as well as giant,firefly?
- Backport changed from giant,firefly to giant,firefly,dumpling
- Status changed from Pending Backport to Resolved
- Description updated (diff)
- Related to Bug #46437: Admin Socket leaves behind .asok files after daemons (ex: RGW) shut down gracefully added
Also available in: Atom
PDF