-1E - RADOS Gateway refactor into library internal APIs

scuttlemonkey    ok, I just refreshed the feed    10:23
scuttlemonkey    it may reload once more    10:23
fghaas    joshd: yes indeed    10:23
*** xiaoxi has quit IRC    10:24
joshd    fghaas: lmb: I'm curious what async-replicated rbd would solve beyond sending rbd differentials between snapshots to another cluster    10:25
joshd    fghaas: lmb: if you send diffs frequently they start looking pretty similar    10:26
fghaas    joshd: well now that you mention it, shipping snapshot diffs might actually be better than streaming replication    10:28
scuttlemonkey    both video feeds should be live now    10:28
fghaas    because it would allow people to have a backup site trail x hours behind the live site on purpose    10:28
scuttlemonkey    we'll get the sessions started up again in another 2 mins or so    10:28
fghaas    joshd: is snapshot diff shipping coming for dumpling?    10:28
gregaf    wido: if you want to talk, the Google+ app supports hangouts and should open automatically if you go to the page in the browser; if you're just watching you can open it on the youtube site    10:29
wido    gregaf: I'm in already :)    10:29
joshd    fghaas: I've got good news for you then, diff import/export are already there in cuttlefish    10:29
gregaf    good good    10:30
fghaas    joshd: I do know that, but is there a daemon that will actually ship offsite or do people still have to handcraft that?    10:30
rturk    Hey all - #ceph-summit2 is for the second track    10:30
rturk    head over there if you're in track 2 :)    10:30
joshd    fghaas: shipping isn't handled by anything right now    10:31
rturk    and if you want to participate in the discussion, send scuttlemonkey a ping    10:31
fghaas    (joshd, in case you're wondering why lmb isn't responding, he stepped out for a bit, will probably throw his comments in when he returns)    10:31
*** nwat has joined #ceph-summit1    10:31
*** maclin has quit IRC    10:32
joshd    fghaas: perhaps the rgw-geo agent could do similar things for rbd, I haven't thought much about it yet    10:32
*** nwat has quit IRC    10:32
fghaas    joshd: ok. so what people would have to do is write a cron job that exports a diff, ships it, and then applies it on the other end? that's ok for testing, but people might dislike it for the real deal    10:32
paravoid    that's not accurate    10:33
paravoid    rgw isn't CGI, it's FastCGI    10:33
paravoid    and FastCGI isn't an old, deprecated way of running apps    10:33
paravoid    and fairly high performant, better than Python for sure    10:33
*** sjustlaptop has quit IRC    10:34
joshd    fghaas: I agree the management of diffs could certainly be made easier, it sounds like you might have some suggestions. I haven't thought about it much yet    10:34
joshd    fghaas: I just wanted the basic tools in place first for cuttlefish    10:35
*** Ryan_Lane has joined #ceph-summit1    10:36
fghaas    sounds a bit like the old Postgres WAL shipping. Which isn't quite as elegant as streaming, low-level RADOS replication, but probably a far better option for Glance/Cinder than an alternative where Glance images are stored as Swift objects via rgw (no objection to rgw at all, but in this case I'd greatly prefer native RBD)    10:37
fghaas    at any rate, joshd, our discussion is going OT for this channel, better switch this to #ceph :)    10:37
loicd    there is a project to move the swift API out of the swift implementation. That was discussed during the openstack summit but I don't have a link handy.    10:42
*** cdl has joined #ceph-summit1    10:42
*** jtran has left #ceph-summit1    10:47
gregaf    we already have so many logs for RGW, and coordinating consumers can be pretty annoying    10:47
scuttlemonkey    need an aggregator log tool like juju's debug-log or something :P    10:48
gregaf    sagewk was talking about keeping an event log and I don't think that's much less appealing to external services than a librgw that you can embed in your app to do all the real work ;)    10:49
loicd    joshd: do you remember about it ? ( the swift library ). Although it's already implemented, it might save time on upgrading and chasing the changes. But I don't really know anything there ;-)    10:52
*** Ryan_Lane has quit IRC    10:53
joshd    loicd: yeah, I talked to some swift folks about it, and first plan is pluggable fs backends for gluster/zfs on the backend servers, and later maybe pluggable higher-level backends (like ceph) beneath the proxy server later    10:54
paravoid    a low-level librgw library that doesn't include the HTTP bits sounds useful in general    10:54
gregaf    sagewk: not sure how behind this is your discussion, but the problem with get and put is all the streaming of those requests, right?    10:54
paravoid    hooking the logic directly into applications    10:54
sagewk    that's the devil in the details, yeah    10:54
paravoid    while also having radosgw on top of that    10:54
scuttlemonkey    (the youtube feed is about 1m behind the hangout)    10:54
paravoid    for example, we could use librgw PHP bindings for mediawiki to access directly the Ceph cluster without a gateway    10:55
joshd    loicd: it might be useful for keeping up with the apis, like new keystone etc, but probably a lot easier with librgw    10:55
loicd    it looks like it could leverage librgw indeed :-)    10:56
joshd    loicd: but the swift guys basically aren't going to do the work to make the proxy backends pluggable (and certainly not soon), so we'd need to do that as well if we wanted to use it    10:56
loicd    ah    10:56
paravoid    youtube is flaky dammit    10:58
scuttlemonkey    yeah, a bit    10:58
loicd    youtube    10:59
loicd    darn    10:59
paravoid    don't assume a MQ    11:01
paravoid    maybe a fork/exec or a unix socket (for performance)    11:01
gregaf    I'd feel better about a unix socket than a blind fork, yikes    11:04
paravoid    could the georeplication logs be reused for this?    11:04
gregaf    reused for which?    11:06
paravoid    the swift API has object expiration already    11:07
paravoid    11:07
paravoid    X-Delete-At & X-Delete-After    11:07
paravoid    but this isn't LRU unfortunately    11:08
paravoid    I wouldn't know about S3, sorry :)    11:08
*** Aaron|away has joined #ceph-summit1    11:10
gregaf    wow, watching the etherpad edits in real-time and then hearing the conversation a minute later is very confusing    11:10
Aaron|away    swift temp urls :)    11:10
*** Aaron|away is now known as AaronSchulz    11:10
* loicd lost connectivity, changing location    11:10
paravoid    AaronSchulz: faster than you!    11:11
*** loicd has quit IRC    11:11
sagewk    what i stoken server?    11:11
michael_dreamhost    with s3 you pull acess tokens for mobile apps say    11:12
AaronSchulz    paravoid: I was listing but not on IRC    11:12
paravoid    gregaf: better than having the stream be interrupted every third minute or so    11:13
gregaf    so we're talking about token servers in terms of something like Keystone, right?    11:13
paravoid    at least I can keep up with the pad :)    11:13
joao    I propose waiting for everyone on the hangout to wait until able to sync IRC with video :p    11:13