libvirt: fix definition for rbd params/sources/etc
#3 Updated by Sage Weil over 7 years ago
The virtual disk was added to libvirt in 036ad5052b43fe9f0d197e89fd16715950408e1d.
It only lets you specify server hosts, nothing else. Librados wants
- client name/id
- conf path (optional)
- list of conf key/value pairs
These can be smooshed into one config string if need be (this is what qemu takes, actually). For example, "id=foo:conf=/path/to/conf:otheroption=that:foo=bar".
In the end, the <host> items translate into an option like "mon_host=host1,host2,host3".
#4 Updated by Wido den Hollander over 7 years ago
Idea from gregaf was to "abuse" the name attribute to do so, have to look into this.
We should check this with the libvirt guys and see what we can do.
Their idea was to have a general format which can be used for Sheepdog, Ceph and NBD and other future network storage which would come up. Instead for writing a implementation for every project.Some docs/discussing about the current format:
Main thing is that we want to push some key=values down to Qemu so that can do down to librbd without manipulating or interpreting it too much.
Some old hack in libvirt allowed virtual disks like:
<disk type='virtual' device='disk'> <driver name='qemu' type='rbd' cache='writeback'/> <source path='rbd:rbd/beta:conf=/etc/ceph/ceph.conf'/> <target dev='vda' bus='virtio'/> </disk>
Works like a charm, but not what libvirt wants :)
#5 Updated by Wido den Hollander over 7 years ago
You can abuse the current libvirt implementation though.
<disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source protocol='rbd' name='rbd/beta:conf=/etc/ceph/ceph.conf:id=admin'> <host name='monitor.ceph.widodh.nl' port='6789'/> <host name='monitor-sec.ceph.widodh.nl' port='6789'/> <host name='monitor-third.ceph.widodh.nl' port='6789'/> </source> <target dev='vda' bus='virtio'/> </disk>
That works for me and results in a string to Qemu:
CEPH_ARGS=-m monitor.ceph.widodh.nl:6789,monitor-sec.ceph.widodh.nl:6789,monitor-third.ceph.widodh.nl:6789 ..... -drive file=rbd:rbd/beta:conf=/etc/ceph/ceph.conf:id=admin,if=none,id=drive-virtio-disk0,boot=on,format=raw,cache=writeback
The benefit of passing down a custom string through libvirt is that we never have to wait for them to implement a new scheme.
When RBD starts to become mainstream people will stick to the libvirt version supplied by their distro. If that means they'll have to wait for 2 years to get a new libvirt on their machines, that's a long time.
Right now you can simply put everything you need in the "name" attribute, but we'll have to check with the libvirt guys how they think about that.
#6 Updated by Sage Weil over 7 years ago
Hmm yeah, it's at least doable. I just sent an email to libvir-list asking about a more generic syntax for options... that'll let us eventually support using kernel rbd driver as well (in a reasonably non-hacky way). Seem ok? (Can probably move discussion to the list..)