Feature #6228: image name metavariable - rbd - Ceph

Feature #6228

image name metavariable

Added by Mark Nelson about 10 years ago. Updated over 8 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Adam Crume

Target version:

% Done:

Source:

Community (dev)

Tags:

Backport:

giant,firefly,dumpling

Reviewed:

Affected Versions:

Pull request ID:

Description

User description¶

Symptoms:

When a VM has two RBD volumes attached, the admin socket only access one of them.

Diagnostic:

Each RBD volume has its own admin socket but their name conflict and only one of them remains.

Workaround:

No workaround.

Fix:

The new meta variable $cctid is defined and the conflicting

[client]
    admin socket = /var/run/ceph/rbd-$pid.asok

can be replaced by

[client]
    admin socket = /var/run/ceph/rbd-$pid-$cctid.asok

Description¶

From Mike Dawson:

There are currently a few metavariables available for use in ceph.conf:
http://ceph.com/docs/master/rados/configuration/ceph-conf/#metavariables

In addition to those listed in that document, $pid was added in Bobtail. That allows me to get RBD admin sockets for libvirt/qemu guests by specifying

[client]
    admin socket = /var/run/ceph/rbd-$pid.asok

in /etc/ceph/ceph.conf. That works well for VMs with a single volume, but it does not work for VMs with multiple volumes. With multiple volumes, the admin only lists data on the last volume specified, leaving me unable to monitor stats on any volumes other than the last.

I imagine you could specify a unique admin socket for each volume on the command-line that libvirt uses to spawn qemu, but that seems like a pain.

Is it possible instead to add a metavariable that would expand to the image name? This way, my ceph.conf would look like

[client]
    admin socket = /var/run/ceph/$imagename.asok

and I would end up with admin sockets like:

/var/run/ceph/rbd-volume-0025b693-6230-443e-ad6a-3aa43efdd648.asok
/var/run/ceph/rbd-volume-009228f3-b8d7-4f3c-8458-0cdbd4809dc6.asok

# rbd -p volumes ls
volume-0025b693-6230-443e-ad6a-3aa43efdd648
volume-009228f3-b8d7-4f3c-8458-0cdbd4809dc6

I'd be happy to provide a patch if someone could give me sufficient guidance on how/where to add this functionality.

Related issues

Associated revisions

Revision bb45621c (diff)
Added by Adam Crume about 9 years ago

common: Add cctid meta variable

Fixes: #6228
Signed-off-by: Adam Crume <adamcrume@gmail.com>

Revision 060a5b14 (diff)
Added by Adam Crume almost 9 years ago

common: Add cctid meta variable

Fixes: #6228
Signed-off-by: Adam Crume <adamcrume@gmail.com>
(cherry picked from commit bb45621cb117131707a85154292a3b3cdd1c662a)

Revision fe705c8f (diff)
Added by Adam Crume almost 9 years ago

common: Add cctid meta variable

Fixes: #6228
Signed-off-by: Adam Crume <adamcrume@gmail.com>
(cherry picked from commit bb45621cb117131707a85154292a3b3cdd1c662a)

Revision 6e3b6f0e (diff)
Added by Adam Crume almost 9 years ago

common: Add cctid meta variable

Fixes: #6228
Signed-off-by: Adam Crume <adamcrume@gmail.com>
(cherry picked from commit bb45621cb117131707a85154292a3b3cdd1c662a)

History

#1 Updated by Adam Crume about 9 years ago

The code would be in src/common.cc. I don't think it would be as simple as just adding another variable, because all of the existing variables are one-per-process, and a process can have multiple images open. The metavariable code would have to be made context-aware.

#2 Updated by Sage Weil about 9 years ago

It's per-cct. We can probably add some methods like

cct->set_meta("image", "foo");

that would populate a map<string,string> and rerun the function that does the substitutions. I think part of the challenge will be making that happen early in the librbd setup, before the admin socket file is set up (probably before common_init() is called?). Should be doable, I think...

#3 Updated by Sage Weil about 9 years ago

...which is hard because librbd layers on top of an existing librados ioctx, which has already long been initialized. We can't even introduce our own default values for anything.

Perhaps the thing to do is make the asok init fail/skip creating the file is there is a metavariable still present (i.e., $ appears in the string) and make the librbd explicit fill it in and then re-trigger something on the CephContext? Something like reopen_logs(), but slightly more general to also include the asok file.

#4 Updated by Adam Crume about 9 years ago

Status changed from New to In Progress
Assignee set to Adam Crume

#5 Updated by Adam Crume about 9 years ago

This assumes that each image has its own cct, but a process could have multiple images open in one cct. (In fact, copy2 requires two open images.) Would we say that we don't support the image metavariable in that case?

What if an image is never opened? You'd never have a .asok.

Should the snap name be included? If so, should the metavariable have a different name?

I'd also like to point out that qemu opens and closes images twice, for some reason, just to make things more complicated.

#6 Updated by Adam Crume about 9 years ago

It's not perfect, but we could add a cctid variable so users could specify something like "admin socket = /var/run/ceph/rbd-$pid-$cctid.asok". This ensures at least that the ccts don't stomp on each others' files, although you'd still have to manually figure out which .asok is for which image.