Ceph

The virtual disk was added to libvirt in 036ad5052b43fe9f0d197e89fd16715950408e1d.

It only lets you specify server hosts, nothing else. Librados wants

- client name/id
- conf path (optional)
- list of conf key/value pairs

These can be smooshed into one config string if need be (this is what qemu takes, actually). For example, "id=foo:conf=/path/to/conf:otheroption=that:foo=bar".

In the end, the <host> items translate into an option like "mon_host=host1,host2,host3".

Idea from gregaf was to "abuse" the name attribute to do so, have to look into this.

We should check this with the libvirt guys and see what we can do.

Their idea was to have a general format which can be used for Sheepdog, Ceph and NBD and other future network storage which would come up. Instead for writing a implementation for every project.

• http://libvirt.org/formatdomain.html
• http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/1323

Main thing is that we want to push some key=values down to Qemu so that can do down to librbd without manipulating or interpreting it too much.

Some old hack in libvirt allowed virtual disks like:

    <disk type='virtual' device='disk'>
      <driver name='qemu' type='rbd' cache='writeback'/>
      <source path='rbd:rbd/beta:conf=/etc/ceph/ceph.conf'/>
      <target dev='vda' bus='virtio'/>
    </disk>

Works like a charm, but not what libvirt wants :)

You can abuse the current libvirt implementation though.

    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source protocol='rbd' name='rbd/beta:conf=/etc/ceph/ceph.conf:id=admin'>
        <host name='monitor.ceph.widodh.nl' port='6789'/>
        <host name='monitor-sec.ceph.widodh.nl' port='6789'/>
        <host name='monitor-third.ceph.widodh.nl' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
    </disk>

That works for me and results in a string to Qemu:

CEPH_ARGS=-m monitor.ceph.widodh.nl:6789,monitor-sec.ceph.widodh.nl:6789,monitor-third.ceph.widodh.nl:6789 ..... -drive file=rbd:rbd/beta:conf=/etc/ceph/ceph.conf:id=admin,if=none,id=drive-virtio-disk0,boot=on,format=raw,cache=writeback

The benefit of passing down a custom string through libvirt is that we never have to wait for them to implement a new scheme.

When RBD starts to become mainstream people will stick to the libvirt version supplied by their distro. If that means they'll have to wait for 2 years to get a new libvirt on their machines, that's a long time.

Right now you can simply put everything you need in the "name" attribute, but we'll have to check with the libvirt guys how they think about that.

Hmm yeah, it's at least doable. I just sent an email to libvir-list asking about a more generic syntax for options... that'll let us eventually support using kernel rbd driver as well (in a reasonably non-hacky way). Seem ok? (Can probably move discussion to the list..)

Saw the e-mail, let's keep the discussion there to make sure everyone involved is up to date.

We could use this issue for some internal work delegation.

I wonder if we can harass someone on IRC to get some partial 'sure, whatever' before we go write patches..

Or Wido, maybe you can follow up on the list and see if someone takes notice?