Ceph

I'm opening this bug on the request of gregsfortytwo, who was very helpful in a brief discussion on IRC (#ceph-devel).

Referring to http://ceph.com/docs/master/rados/operations/cache-tiering/

This seems to imply that the Objector can request cold data directly from the backing tier. Is this the case?

The other subject I'm not clear on is write requests to a write-back cache. Do the writes get forwarded to the backing tier, does the Objector go directly to the backing tier, or does the data get moved to the cache tier and then the write is applied there?

Below is the full IRC log, including some points to be clarified and some answers:

2:30 < pingu_> Hi, #ceph doesn't seem to have some answers I'd like on cache tiering so I'll ask here.
22:30 < pingu_> With cache tiering -- if you're operating in writeback mode and you modify some data that is currently inactive (residing in cold storage), does the data first get copied to the cache tier and then updated?
22:30 < pingu_> Or is it updated in the overlayed (cold) pool?
22:31 < pingu_> Also, when operating in readonly mode, and you get a cache miss, is the data first shuffled to the cache tier and then served to the requesting client?
22:31 < pingu_> Or is it served directly from the cold tier and scheduled to be promoted based on heuristics?
22:31 < gregsfortytwo> in hammer it's all promoted first
22:32 < pingu_> ah, and that's changing?
22:32 < gregsfortytwo> in infernalis (I think?) the reads can be proxied, and some of the writes can
22:32 < gregsfortytwo> by the cache pool OSD that would serve the data if it were in cache
22:32 < gregsfortytwo> and the data doesn't get promoted unless it gets sufficiently hot
22:32 < pingu_> or unless it's a write, right?
22:33 < gregsfortytwo> I'm not sure how configurable "sufficiently hot" is, but I think it means two regular accesses without the cache daemon wanting to evict the metadata
22:33 < gregsfortytwo> writes will generally require copy-up, but some of them can be proxied as well
22:33 < gregsfortytwo> not sure if that code is in the infernalis code base though; this is all in the RADOS team
22:34 < pingu_> Yeah, pretty sure that's tweakable with the hitset options.
22:34 < pingu_> Whilst I have you're attention.
22:34 < pingu_> Do you know why readonly mode simply says "no writes"?
22:34 < gregsfortytwo> nope
22:34 < pingu_> It seems like it would have been easy to add eventual consistency
22:34 < gregsfortytwo> oh
22:34 < pingu_> but I'm sure there's some corner case there
22:34 < gregsfortytwo> yeah, no, that's not Ceph's model and we really can't implement it
22:35 < gregsfortytwo> none of the data tracking structures or algorithms are capable of eventual consistency
22:35 < gregsfortytwo> in addition to it just being against the Ceph philosophy
22:35 < gregsfortytwo> read-only is read-only
22:35 < pingu_> okay, thanks.
22:35 < gregsfortytwo> you can write to the base pool and the read-only pool will pick it up eventually, that's about as eventual-consistency as it's going to get ;)
22:35 < pingu_> that was most helpful
22:36 < pingu_> gregsfortytwo: sure, that's what I meant actually
22:36 < gregsfortytwo> oh, I *think* you can do that?
22:36 < pingu_> I was just thinking that when you write to the base pool, the read only pools are told to drop those things from their caches
22:36 < pingu_> and you don't have to wait for eviction
22:36 < gregsfortytwo> don't think so
22:37 < pingu_> Yeah, pretty sure you can't currently, but that seemed easy to implement is all and I was wondering why not.
22:37 < gregsfortytwo> but read-only pools aren't as well tested or supported as the more normal ones so I'm not sure exactly what state they're in
22:37 < gregsfortytwo> if not available right now it's probably just because configuring the clients to ignore the caching redirects isn't set up and would be a pain
22:37 < pingu_> okay
22:38 < pingu_> One last thing!
22:38 < pingu_> the "Objecter", is that client-side code?
22:38 < pingu_> as in, there's nothing proxying at the network layer between clients and the various tiers, is there?
22:39 < pingu_> The clients are just directed to the hot or cold tier based on OSD maps of some sort? If it's in the cold tier does the client know to go straight there?
22:39 < pingu_> Or does it always hit the cache tier and then get some kind of a redirect on miss?
22:40 < gregsfortytwo> Objecter is client-side, yes
22:40 < gregsfortytwo> part of the internal library, and it handles redirecting clients to the hot tier
22:40 < gregsfortytwo> clients don't touch the cold tier if there's a caching pool at this time
22:41 < pingu_> Oh. That's a surprise.
22:41 < pingu_> The diagrams are kind of a lie, then.
22:41 < gregsfortytwo> there are discussions about sending clients a redirect to the cold tier in some cases, but the consistency model of doing that is very difficult
22:41 < gregsfortytwo> which diagrams?
22:41 < pingu_> http://ceph.com/docs/master/rados/operations/cache-tiering/
22:42 < pingu_> first one there, has an arrow from Objecter to Storage Tier
22:42 < gregsfortytwo> huh
22:42 < gregsfortytwo> we've certainly planned to enable that
22:42 < gregsfortytwo> maybe it does that right now if you've got a read-only tier and try to write to the base pool
22:43 < gregsfortytwo> or maybe the diagram is just wrong, can you file a doc bug so somebody can check and get it fixed?
22:43 < pingu_> So currently if there's a cache miss, things are just streamed through the cache OSD from the storage one?
22:43 < pingu_> and then later promoted if that's the will of the tiering agent