Ceph

h1. Ceph User Committee meeting 2014-05-02

h3. Executive summary

The agenda was:

* Elections

* RedHat and Inktank

* CephFS

* Meetings

Action: The Ceph User Committee will express the need for a Ceph Foundation, from the user perspective

Action: Patrick McGarry will organize a RedHat acquisition meeting in two weeks time

Note: Patrick McGarry participated in the meeting and answered questions, as can be read from the logs below. The executive summary  focuses on the points raised by users instead of attempting to summarize the dialog.

h3. Elections

It will happen this month, details here : https://wiki.ceph.com/Community/2014-04_Ceph_User_Committee_Elections

h3. RedHat and Inktank

Positive: stewardship of other projects seem fine.

Concern: support for non RHEL OS

Positive: better support for Fedora

Concern: the project is sold to RedHat, engineers and trademark, the people who were in charge now answer to someone else

Hope: inifiniband support boost from RedHat

Positive: RedHat has experience with maintaining software production ready and supporting customers

Concern: what does it mean from the point of view of Inktank customers

Positive: greater potential for development gains between ceph and kvm

Concern: the [[Foundation|Ceph foundation]] becomes more necessary than ever, to establish a diverse governance, will RedHat agree to it ?

Concern: Sage had the skills and was empowered to be the Ceph benevolent dictator for life. The skill remains but he has less power over the project.

Clarification: Inktank acquisition by RedHat should not be confused with MySQL acquisition by Oracle. The copyright is intentionaly fragmented and cannot be sold.

Feedback: Cloudwatt management reacted positively to the acquisition.

Positive: Calamari will be published under a Free Software license

Confusion: what does it mean for GlusterFS, really ? Features ? Sale points ? Development roadmap ? What Inktank products / services, training will remain ? etc.

Concern: can Ceph, as a software, be reasonably independant from the service provider side of Inktank / RedHat ?

Concern: who coordinates the development, roadmap, feature list ? The Ceph Foundation or RedHat ?

Concern: users must apply pressure for the Ceph Foundation to happen, RedHat has little incentive to agree to it spontaneously.

Action: we, users, should express our desire for a Ceph Foundation, with testimonials collected from various people

Concern: should things go bad and a fork becomes necessary, all the driving forces of the project are currently under Inktank / RedHat influence

Comparison: Qumranet acquisition is perceived to have been beneficial to KVM

Concern: some Gluster, Inc customers were unsatisfied after the RedHat acquisition, could it happen to Inktank customers also ?

Action: Patrick McGarry will organize a RedHat acquisition meeting in two weeks time

h3. CephFS

Use case: project to replace a 40TB cluster used to host mirrors for distributions (binary data packages, iso, tarballs), delivering more than 1Gb/s and less than 4Gb/s

* http://dmsimard.com/wp-content/uploa...or_logical.jpg

* suggestions to use object instead of CephFS, with adhoc software

* if these "webservers"  were openvz containers, having its datastore on cephfs

* if front servers were openvz containers, they could be HA-managed

* the only blocking factor is CephFS not being production ready: mostly the active-active MDS scenario and the dynamic subtree partioning that was unstable

* the plan is to deploy with puppet

* deduplication would help

Use case: OVH.com does something with Ceph (CephFS ?) : https://twitter.com/olesovhcom/statu...82909729763328

Use case: French academic community meeting discussed how CephFS could be used (no record) http://www.capitoul.org/ProgrammeReunion20140424

h3. Meetups

All meetups https://wiki.ceph.com/Community/Meetups

May 13th, Atlanta : http://openstacksummitmay2014atlanta...e#.U2PrHuZdW6w

May 11th, Atlanta : http://www.meetup.com/Ceph-in-Atlanta/

May 9th, Berlin : http://www.meetup.com/Ceph-Berlin/events/179186672/

h3. Log

<pre>

<loicd> Welcome to the Ceph User Committee meeting #2 ! https://wiki.ceph.com/Community/Meetings#Proposed_topics:

<loicd> scuttlemonkey: will join a little late

<Vacum> Hi :)

<janos_> ooh #2!

<loicd> ahah

<janos_> do we have to wear red-colored hats?

<janos_> ;)

<loicd> I propose we get the easy stuff / boring things out of the way first and give a chance to the people who are late to join

<loicd> janos_: not yet I hope

<janos_> lol

<scuttlemonkey> I'm here! :)

<loicd> First topic : the elections     

<loicd> as promised I'll send a mail later today to setup the elections of the Ceph User Committee head

<loicd> it will be interesting in the new context ;-)

<loicd> I will apply and mourgaya proposed to apply too

<loicd> is there anyone else interested ?

-*- janos_ keeps his hand down

<loicd> the idea is that you spend ~8 hours a week on average caring for the ceph user committee. It's not much but still ;-)

<loicd> mourgaya: are you still on board for this ?

<mourgaya> yes!

<loicd> cool

<loicd> now to more fun things

<loicd> redhat and inktank, what do people think ?

-*- loicd tend to be a kill joy and won't speak first ;-)

<janos_> i think it's good. RH's stewardship of other projects seems to have been good

<janos_> not overbearing

<lesserevil> re: inktank->redhat: optimistically cautious

<Serbitar> my conern as others have raised is the ability to get support for non rhel OS

<janos_> plus i think now i'll get the f20 builds i've been dreaming of

<janos_> ;)

<kevincox> I think that it will be good for the project.

<janos_> yeah i can understand the support concerns for other distro's

<scuttlemonkey> I know I'm inside the beast, but I think it's a good move.  However, it may be a bit of a paradigm shift in 

long-term planning for things like foundation

<loicd> Serbitar: do we know how much time redhat actually invests in supporting kvm for other os (for instance) ?

<Vacum> a bit surprising. during the Frankfurt Ceph Day, the general statement from Inktank and Ceph was "we won't sell out".

 at least it sounded like it

<pressureman> i hope that infiniband support will get a boost from redhat

<Serbitar> loicd: i do not

<scuttlemonkey> fwiw I know sage is working very hard to ensure that support for non-rhel setups is strong

<loicd> Vacum: it can be percieved as a sell out indeed.

<mourgaya> ceph can have  the benefits of redhat  landing production!

<scuttlemonkey> and for now inktank is still selling and supporting ubuntu/suse

<Vacum> I'm a bit concerned about the transistion peroid, also from a commercial support view

<loicd> Vacum: how do you mean ?

<Vacum> scuttlemonkey: "for now" isn't really something a business can rely on when it comes to setting up petabyte storage

<scuttlemonkey> Vacum: I absolutely agree

<scuttlemonkey> it's just hard to answer definitively as the "how do all the bits and pieces get merged" discussions are still ongoing

<scuttlemonkey> all I can give you is intent

<loicd> scuttlemonkey: this is reassuring and there does not seem to be a risk that other platforms support is dropped any time soon. 

I think people express a concern in the long term.

<janos_> i'm excited about the greater potential for development gains between ceph and kvm

<loicd> janos_: I did not think about that, you're correct !

<Vacum> loicd: we have a 12 month pre-production support that will run another 8 months. and we were planning on contracting the 

enterprise support. now its totally open if such a thing will be available in the (near) future - and to which conditions

<scuttlemonkey> vacum: you're ubuntu I'm assuming?

<Vacum> scuttlemonkey: debian

<scuttlemonkey> ahh

<loicd> Vacum: could you spell the name of your company for the record ? Unless it's confidential of course ;-)

<Vacum> loicd: I can spell it per PM :)

<loicd> now that redhat is there, the foundation becomes more necessary than ever

<Vacum> I do see a plus on the whole thing from a commercial perspective though. RH does have a long history in providing enterprise 

support and they know it all. inktank can benefit from that

<loicd> in the past, as a user, I felt confident that Sage could be a benevolent dictator in the broad sense of the term, not just 

technical. Now that redhat is involved, there needs to be a diverse governance of some kind.

<mourgaya> is redhat keeping the inktank support team, and there reactivity?

<loicd> Vacum: +1

<scuttlemonkey> loicd: that's my take as well, but there is mixed opinion from the folks involved....so I'm looking forward to the discussions

<amichel> Question about the repositories for ubuntu/debian. I'm doing a deploy on 14.04 trusty and the ceph-extras repo doesn't seem to have trusty packages. Is ceph-extras not needed on 14.04 or is there a trick I'm missing?

<Vacum> loicd: I totally agree. "Ceph" as a brand (and I didn't use trademark on purpose!) should not fall into the hand of a company

<scuttlemonkey> mourgaya: Inktank is remaining intact as an org until we can ensure that the transition wont change support response

<loicd> mourgaya: we don't know. But as a group of users I think we should see the broader consequences. In a few years from now, if all 

goes well, ceph will be some kind of kvm. Widely spread and adopted. Is this what we would like ? Would our use cases be satisfied by such an outcome ?

<loicd> amichel: we're having a meeting (ceph user committee). Do you mind if we postpone the answer for another 45 minutes ?

<amichel> No problem at all, I didn't realize

<Vacum> from a brand (and even trademark) perspective, look at MySQL. They sold to Sun, which was kind of cool. and now its at Oracle...

<nobody18188181> Vacum: But now MariaDB is taking over ;)

<loicd> dmsimard: what  does iweb think of this move ?

<loicd> Vacum: MySQL copyright was sold to oracle. That cannot happen with Ceph.

<Vacum> nobody18188181: yes, Maria brings a lot of new stuff (no wonder, coming from Monty). but just look at the channel's activities...

<Vacum> loicd: the trademark was sold too

<loicd> the copyright is intentionaly fragmented

<mourgaya> good!

<nobody18188181> What do they do in their channel?

<dmsimard> I can speak for myself, not quite on behalf of iWeb as is - I'm happy for Inktank and that Calamari will be open sourced.

 I am really curious as to what will happen with Gluster since, to me, Ceph is a natural competitor to gluster.

<loicd> it was a wise decision.

<janos_> i don't imagine much will happen to gluster

<loicd> dmsimard: +1

<Vacum> nobody18188181: I mean IRC channels. compare the activity of both

<janos_> RH will likely be happy selling support for both

<mourgaya> radhat is now  the leader of the futur of storage :-)

<nobody18188181> Ah, I havent been to the maria or mysql channels so I cant speak on them

<dmsimard> I am not super familiar with Gluster but does it do things that Ceph does not do ?

<dmsimard> That's kind of where I am getting at

<loicd> My company ( Cloudwatt ) has reacted positively to the announcement.

<loicd> The marketing director came to me and showed the planned cooperation with RedHat. He said : "we'll add a Ceph line there". And that was it.

<Serbitar> dmsimard: i guess it would be that cepgh has more functionality than gluster, with block object and file stores vs glusters' file store

<Serbitar> evne though csphfs isnt commercially supported yet

<loicd> Is anyone around here actually using gluster ?

<dmsimard> So from a Redhat perspective, do you continue to develop both ? Do you focus your efforts on Ceph ? This is what I am curious to see how it plays out.

<nobody18188181> loicd: Per recommendation of a friend I'm trying to use ceph (gluster was going to be my first choice); but if i cant get ceph working then I'm going to have to try that.

<Vacum> I'm a bit on the cautious side. On the RH announcement they are talking a lot about Inktank's "products". Do they mean the services with that. Or Ceph itself?

<loicd> nobody18188181: I see

<nobody18188181> loicd: I chose ceph because a good friend of mine indicated to me that ceph is vastly superior in performance compared to gluster; so of course that part wins me over.

<scuttlemonkey> Vacum: the Inktank "product" is "Inktank Ceph Enterprise"...which is Calamari + Support/Services...there is also training/consulting

<loicd> We can only speculate and hope for the best. In terms of timeframe, I bet we'll know where we stand in a year or so.

<Vacum> also, during Frankfurt Ceph Day, Sage talked about keeping Ceph as the product/solution and Inktank as service provider seperate. Is that even possible with RH?

<loicd> Vacum: with a foundation it is.

<lesserevil> loicd: +1

<Vacum> loicd: But then, who is the "coordinator" of Ceph's development? The Foundation, or RH?

<mo-> as somebody that's been trying to tell people that Ceph is worth a look (or two), I find it a BIG plus to be able to add that it is a RH supported solution now

<Vacum> Who, authoritively, will decide what goes in as a feature and what not?

<loicd> Will RedHat agree to a foundation holding the trademark, I would not be on it. But that depends on us (users) and the developers from the community.

<janos_> mo- good point when you have to make that pitch

<scuttlemonkey> Vacum: the idea would be foundation as a central clearinghouse for development...but each contributing org would have their own plans/roadmap (including RH)

<scuttlemonkey> if such a foundation were to occur, Sage would still be BDFL and decide what goes in, and how

<Vacum> so the "upstream" of everything would be The Foundation

<scuttlemonkey> yeah

<Vacum> that would be nice

<scuttlemonkey> that's my hope

<Vacum> mine too :)

<nobody18188181> ok i found a bug how can i report it quickly?

<scuttlemonkey> all depends on how RH sees the future

<loicd> Vacum: sage must be the benevolent dictator for life. At least I believe it's necessary because of the Ceph dymanic. A personal opinion based on observation and betting that what happened in the past will work in the future ;-)

<loicd> If I was RedHat I would not allow the creation of a foundation. Unless there is significant pressure from the community.

<Vacum> exactly

<scuttlemonkey> yeah

<scuttlemonkey> and to be fair there are a number of great single-vendor FOSS projects

<loicd> I propose that we voice, loud and clear, what we would like to see in a foundation. And why we think it is necessary.

<scuttlemonkey> so it'll be an interesting discussion at least :)

<loicd> scuttlemonkey: right :-)

<scuttlemonkey> please do

<Vacum> +1

<scuttlemonkey> I have spent several months thinking about a foundation

<xarses> should we create a petition?

<scuttlemonkey> so I'd love to have new information injected into those thoughts

<scuttlemonkey> xarses: not necessary...Sage and I are already on the path

<loicd> https://wiki.ceph.com/Development/Foundation has your ideas right ?

<scuttlemonkey> you could contribute to the wiki doc though

<mourgaya> foundation does not depend of redhat, ceph is an open source solution  right?

<scuttlemonkey> loicd: only the very highest level brush strokes...but yes

<scuttlemonkey> mourgaya: the point of the foundation would be to hold the trademarks in trust for the community

<loicd> xarses: something that looks like a petition without the controversial tone would be nice

<janos_> an assertion

<scuttlemonkey> without Red Hat's donation of those marks the foundation really can't happen

<scuttlemonkey> loicd: xarses: I propose we just create an "interested parties" section on the foundation doc

<scuttlemonkey> for those who are interested in seeing it happen

<loicd> mourgaya: the dynamic of the project depends on redhat now. And a fork would be most difficult. The idea of a foundation is to make such a fork unecessary, forever, because all interests are represented.

<Vacum> a fork wouldn't have much chances. too much happening in the code at the moment. only if Sage and a few other key devs would create that fork themselves, it would stand a chance

<Vacum> see Maria...

<scuttlemonkey> hehe

<loicd> scuttlemonkey: having a document where people can freely express their thoughts, even if not polished, would be useful

<scuttlemonkey> loicd: so you're thinking a "talk" page in addition to the brainstorm doc?

<loicd> right

<fghaas> um, can I just inject one thought here since scuttlemonkey asked for it: is anyone under the impression that RHT fucked up KVM, post-Qumranet acquisition?

<loicd> xarses: is this something like this you had in mind ?

<janos_> fghaas, not that i can tell

<xarses> loicd: something like that

<mourgaya> how can we have  redhat position about a ceph foundation?

<loicd> fghaas: Inktank can't compare to Qumranet because they had a proprietary software base to begin with. Intkank is a Free Software shop and this is a significant difference.

<scuttlemonkey> mourgaya: I will be sharing that info as we start the discussions

<xdeller> fghaas: absolutely not, except some directions, like state replication, was abandoned

<fghaas> loicd, KVM was always free software. The *management products* around KVM were not. Ceph is free software, Calamari is not. I maintain there's significantly less difference than you think. And yes, RHT would have us believe that RHEV-M is The best Thing Since Sliced Bread™ for a few years, but then OpenStack set them straight

<loicd> fghaas: we can debate this historical thing later ;-)

<loicd> Should we move to more technical topics or someone has more to say about the redhat acquisition ?

<Vacum> its a bit early for outsider to have more insight to talk in-depth about it :)

<loicd> true ;-)

<fghaas> so I'm with janos_ and xdeller here; I think RHT has been a fine steward of KVM, and if they follow *that* precedent then the Ceph user community will rather be very happy with them. But they certainly broke some glass with the Gluster deal

<fghaas> so they better learn the right lessons from their own history :)

<Vacum> perhaps we can trace the RH thing a bit more closely than every 4 weeks?

<loicd> fghaas: did then ?

<loicd> did they ?

-*- loicd knows nothing about the Gluster deal

<loicd> Vacum: how do you mean ?

<fghaas> loicd: oh yeah, there were quite a few Gluster, Inc. customers they pissed off by just not offering GlusterFS support on RHEL, and instead forcing customers to go with RHS if they wanted GlusterFS support

<loicd> ah

<loicd> indeed

<kraken> http://i.imgur.com/bQcbpki.gif

<loicd> dam

<loicd> ahah

<Vacum> loicd: perhaps have a 30 minute "Ceph User Commitee Special" in 2 weeks only for that topic?

<mourgaya> argh!!

<Vacum> not good?

<loicd> Vacum: if you're go to organize this, I'm in !

<Vacum> loicd: ha, who wants to spend 8 hours a week for the Commitee? :D

<janos_> so it sounds like with the RH deal there are two camps with very different concerns - those who use the product as-is puublicly and those with support contracts

<loicd> Vacum: let's discuss this after the meeting.

<scuttlemonkey> loicd: I'm happy to organize ad hoc meetings for this topic as I uncover answers WRT foundation

<janos_> the public crew shouldn't really see anything but general benefit

<janos_> imo

<Vacum> janos_: actually I'm currently in the limbo between both - _because_ of the acquisition

<loicd> scuttlemonkey: ok !

<fghaas> yeah janos_, jftr, I don't think anyone complains about RHT's stewardship of GlusterFS the project

<loicd> fghaas: so you're generally happy about this deal ?

-*- loicd remembers that we should keep some time for the CephFS topic, 20 minutes left ;-)

<janos_> ooh oho there's new stuff to say about cephFS?

<loicd> dmsimard: you had a use case to discuss IIRC ?

<dmsimard> loicd: yeah, I can talk a bit about a use case I have for CephFS

-*- loicd listens

<fghaas> loicd: re your question, I'm all for people striking rich that I like and whose work I deeply respect :)

<janos_> fghaas, haha, yes i agree

<dmsimard> iWeb is a mirror for a lot of open source distributions, some of which are officially recognized mirrors by upstreams - http://mirror.iweb.com/

<loicd> fghaas: :-)

<dmsimard> Being a mirror means having to provide a lot of space, a lot of network throughput

-*- loicd clicks

<loicd> dmsimard: lots as in how much ?

<iggy> aww man, click spam again!

-*- iggy kids

<loicd> iggy: :-D

-*- pvh_sa listens (<-- is a cephfs user, silly me, but I got my reasons)

<dmsimard> Right now we're hovering around 40TB of data

<loicd> all on CephFS ?

<dmsimard> No, not on CephFS.

<dmsimard> I wish it could be, though.

<loicd> I should not interupt and let you finish with the use case ;-)

<dmsimard> Right now the data resides on multiple JBODs daisy-chained with a head.

<dmsimard> It's hard to scale effectively, not as highly available as we wish it could be

<xarses> dmsimard: doesn't radosgw swift/S3 API make more sense for that?

<xarses> mabe a web server to make it look like a fs again

<loicd> is there such a thing ?

<xarses> there should be =)

<dmsimard> I don't know if it could be done, some mirrors are different than others - some push, some pull, etc.

<dmsimard> Anyway, I brainstormed about doing this with CephFS and it'd look like this: http://dmsimard.com/wp-content/uploads/2014/04/mirror_logical.jpg

<loicd> when are you planning to deploy this ?

<dmsimard> This provides the ability to easily scale a highly available storage backend, scale the amount of webservers - probably in 1Gbps increments - as more network throughput is required

<dmsimard> Right now all the mirrors are hosted on this single web/storage beast

<dmsimard> Having the setup above would allow us to scale each mirror according to it's own needs, adding mirrors would be simple.

<loicd> I wonder if anyone has done this before. It looks like a natural / simple fit.

<dmsimard> loicd: I know that OVH fairly recently touted they used Ceph for their mirror infrastructure (this was after I brainstormed the above!). I don't know if they use block or CephFS.

<loicd> Would you like to create a CephFS use case page and add yours ?

<mo-> imagine those "webservers" (shouldnt they be FTP/rsync servers?) were openvz containers, having its datastore on cephfs... perfect segway

<Vacum> dmsimard: you will likely want to cache on the front-facing mirror servers nevertheless IMO

<loicd> I've not heard of OVH lately but they are rather secretive about what they do.

<loicd> mo-: did you try this already ?

<dmsimard> loicd: https://twitter.com/olesovhcom/status/433982909729763328

<mo-> no, I was just saying. this doesnt seem very different from the openvz container usecase

<Vacum> dmsimard: I would imagine you have high peaks on the same few files. ie every time a new version is published and all people DL the same .iso?

-*- loicd should live in the 21st century

<loicd> mo-: that makes a lot of sense

<loicd> dmsimard: how much bandwidth is your mirror having ? peak time ?

<dmsimard> mo-: The frontend servers would be very identical indeed, with only the mirror pool subfolder changing - we in fact planned to leverage Openstack (perhaps with Heat) to scale easily.

<loicd> dmsimard: thanks for the link

<mo-> if these front servers were openvz containers, they could be HA-managed as well, no need to manually mess with HA on application-level then

<dmsimard> loicd: Don't have the data on hand, the server on which the server resides has a 4Gbps LACP link, haven't heard of it being maxed. I know it's more than 1Gbps though.

<loicd> ok.

<mongo_> What type of files?

<mongo_> what size?

<dmsimard> mo-: I'm not personally familiar with OpenVZ :( I would need to look into it maybe.

<loicd> mongo_: I would assume mostly iso + tarbals + packages

<dmsimard> mongo_: Linux distribution mirrors, so binary data packages, iso, tarballs

<dmsimard> We're also a mirror for sourceforge so a lot of binary there.

<mo-> its like BSD jails. many systems running on one host with almost zero virtualisation overhead. much more efficient than hardware virtualisation in fact

<mongo_> I just use nginx, I have it try the local file, local peer and if not it grabs it from upstream and saves the file locally

<mongo_> much easier to scale and far less complicated to maintain.

<loicd> Last week I was at http://www.capitoul.org/ProgrammeReunion20140424 and people from R&D in universities were very attracted by CephFS. Mostly for legacy applications.

<dmsimard> mongo_: Yes, of course some caching layer would be involved. The graph I linked earlier is super high-level

<mongo_> you would be better off with radosgw as it has built in geo replication

<loicd> dmsimard: do you feel something is missing from CephFS that would make things easier to setup for this use case ?

<mongo_> ceph-fs is not really ready for prime time right now.

<loicd> mongo_: how would you mirror things without cephfs ?

<dmsimard> loicd: I was able to setup/puppetize and use CephFS fairly easily in my continuous integration infrastructure. What's stopping me is the red light that it's not production ready.

<loicd> you would need to write software

<loicd> ok :-)

<dmsimard> I know what it's mostly the active-active MDS scenario and the dynamic subtree partioning that was most unstable last I heard

<mo-> I would wager that deduplication for cephfs would be a great fit for such a mirror system

<loicd> mo-: +2

<loicd> We have 2 minutes left.

<Vacum> make that "deduplication for rados would be a great fit" :D

<dmsimard> mo-:        Deduplication is great if you have the same data all over the place, this is not my case here though ?

<loicd> I'll announce the next meeting (early june) on the mailing list.

<mourgaya> dmsimard: dmsimard: +1

<loicd> If you're lucky enough to go to Atlanta next week, don't miss the meetup ! http://www.meetup.com/Ceph-in-Atlanta/ :-)

<dmsimard> Lots of iWeb folks going to the summit, not me unfortunately :(

<scuttlemonkey> or the design session!

<loicd> And if you're in Berlin (lucky too, it's a great time to be there) : http://www.meetup.com/Ceph-Berlin/events/179186672/ is an opportunity to meet Ceph people.

<scuttlemonkey> http://openstacksummitmay2014atlanta.sched.org/event/ddecd66323efb0c83baeb1bbc1d9556e#.U2PrHuZdW6w

<scuttlemonkey> that is a mini-CDS for OpenStack-related devel work discussion

<loicd> scuttlemonkey: :-)

<Vacum> btw, when is the next online CDS planned? :)

<scuttlemonkey> Vacum: haven't set a date yet... I was waiting to see what the timetable looked like in a post-firefly release world

<loicd> We're running out of time but we can keep going on #ceph-devel :-)

<loicd> Thank you everyone !

</pre>