Ceph

We've discussed this sort of log-only replica at a few times in the past. It's conceptually simple, but unfortunately a fairly big effort to create and (especially) validate, since it creates a whole new kind of peer which doesn't fit into the normal peering and recovery framework at all. That means major changes to peering to differentiate between them and figure out when you do and don't have all the data you need, major changes to recovery and backfill to make sure "log-only" replicas which get promoted to full actually see all the data they need and can deal with having logs entries for data they don't have, etc etc etc.
None of that is to say it's impossible, but it's a major features rather than something one could hack together in a few days, and the use case for it so far appears sufficiently specialized that it's not become worthwhile.

Err... I'm confused; it looks like we already do pretty much everything it would take to implement this feature. Say, while we're recovering or backfilling an osd, we already push to it objects that are modified, don't we? So all it would take would be a PG state that was like these, but that would NOT push other objects to it. Something like wait_biab, very much like wait_backfill or wait_recovery. What am I missing?

You're missing the ways in which this differs from existing recovery mechanisms:
1) For this state, we would want to keep a log which contains the changes to objects. In all other states, we keep a log that references changed objects (but not the contents of the change).
2) When recovering from an OSD that is partly up-to-date, we have a single line (backfill_thru or whatever it is) we can point to and say "everything prior to this line is up-to-date. With this feature, if we have an old copy of the PG which overlaps with this log, we instead need to recover whatever random objects are in the full data log.
3) Right now, OSDs who have newer data than their peers have a full copy of the data, and backfill works by copying the full object (not just changed extents). We'd have to handle this too.

etc etc etc

None of this makes a feature like this a bad idea nor an infeasible one to implement, but it is sufficiently large that it will require dedicated work and a use case compelling enough to make somebody pay for it. :)

Greg, it looks like you're discussing difficulties related with the more elaborate plans for the log-only replica, which fulfills a similar goal but is quite different from what I suggested.

I've observed on my own cluster an osd (say osd0) start backfilling a PG (say 0.0) onto another osd (say osd1) while a third replica (say osd2) was down, while the PG had objects created, modified and removed. Part-way through the backfill, osd2 came back and osd0 went down, and osd2 could recover from osd1 the changes that had been made while osd2 was down, even though osd1 had been only partially backfilled. Once recovery was completed, it advanced to backfill state and continued backfilling osd1.

Now, while changes were being made to both osd0 and osd1, osd0 was also backfilling osd1. My suggestion amounts to nothing more than introducing a state in which osd0 “starts backfilling” osd1, but doesn't really backfill any objects, sending to osd1 only the (full) objects that it would send because they are modified.

Heck, we even have a global nobackfill state that might accomplish this by keeping PGs in wait_backfill state; it's just not on a per-osd basis. Assuming modified objects are pushed to the to-be-backfilled replicas even in wait_backfill, that is.

Did I get any of the above wrong?

I can't really follow the story you're telling, but even if you are backfilling only changed objects, you still have all of the "I hold a random subset of objects" difficulties later on. More than that, you now need to copy every object being touched to a new OSD before the write can go through.
And modified objects are not pushed to out-of-date OSDs when in wait_backfill. :)