Ceph » rgw

See #3080 for a similar RADOS-level feature. Proposed separately for radosgw as it can offer different tradeoffs, and we may decide to implement one or the other, but not both.

For RadosGW, it seems we could make the gateway log all write operations (needed for billing anyway?), and update the replica by following those logs. Keep a persistent pointer to how much of the usage log has been consumed, in case the replication process crashes. A challenge would be maintaining acls; perhaps the
replicator would need to talk librgw not http.

In the limited radosgw use case, clients must understand that write ordering across objects might not be respected (e.g. put a=1, put b=2, put a=3; replication sees a needs to be replicated, replica sees a=3 before
b=2). The alternative to this would be completely versioned (MVCC) rgw storage, which sort of sounds like what the garbage collection thing already enables.. preserving ordering might not be worth that.

Pasting Yehuda's comments from email (I can't claim to understand them):

Note that we have two kinds of logs (ops logs, and usage logs) and
both are not adequate. The problem with the op logs is that we don't
really care if we lose some info if the process crashes.
Maybe we can piggyback on the bucket index, keeping entries also by
modification time, so that we can fetch all entries in a bucket that
were modified since a specific timestamp. This will require some more
thinking, as it doesn't clear all cases (e.g., object removal).

As far as how complicated this will be, assuming it's along these
lines, it's going to be as complicated as the usage logging and/or the
garbage collection.