Ceph

As a new user who spent hours googling and reading source code to decipher what each tool does, I thought of giving some feedback :)

There is currently a multitude of tools available, all of them doing similar things in a different way, as far as I understand it. Documentation suggests either a combination of mkcephfs and manual steps (e.g. adding an OSD) or Chef. However, it's not even clear from documentation what mkcephfs does internally, or even how each command from those manual steps affects the cluster or config. I was also amazed to see that the alternative, the Chef cookbook, didn't follow those steps but did its own thing.

There's ceph-deploy which anecdotal evidence (mailing lists, blogs) suggests is the replacement of mkcephfs and the way forward but there's virtually no documentation about it. That's #3394, although I'm wondering if that covers changing the rest of the documentation/deployment howtos or is about just documenting the tool.

Then there's ceph-disk-prepare, ceph-disk-activate and ceph-create-keys which I found by accident. I read their source and the Chef cookbook to understand what they do exactly and how it relates to the rest of the manual steps (even the Chef cookbook doesn't use ceph-create-keys but reimplements it internally). #2774 is about ceph-disk-* manpages, but there's nothing suggesting how/if these will be hooked up into the main documentation.

Additionally, there seem to be significant differences on how ceph.conf should be structured, depending on the tool of choice. ceph-disk-* support a very lightweight configuration, while the "adding OSDs" part of the manual suggests adding [osd.NNN] sections. There's little information about [global] "mon hosts" and "mon initial members" and how these are different than [mon.NNN] sections.

The bootstrap-osd key is also completely missing from the documentation, although it's being used by ceph-deploy, ceph-disk-* and Chef. There are other details like those.

And I'm only listing what is officially "supported": the superset of documentation and git tree. I also studied the unofficial puppet recipes and I'm sure Juju does its own thing too :-)

I think it's okay to provide people with multiple options but it'd be nice to:

• List all these ways, how they're different from each other (e.g. the manual steps are these, mkcephfs handles 1, 3, 4; ceph-deploy handles 1-5, ceph-disk-prepare handles 4, 5, 6)
• What are the pros and cons of each (e.g. hotplugging of disks)
• How the configuration needs to be adapted in each case (possibly including a complete ceph.conf :)
• Which one is recommended by the authors and which one is going to be supported in the long-term.

FWIW, we're using puppet and so far I'm leaning towards rewriting the puppet module to use ceph-create-keys/ceph-disk-prepare/ceph-disk-activate. I'm still not sure about my choice or whether they are supported and will be in the long-run :)