Ceph

Isn't the idempotency in that case "clone foo_head -> foo_2 IFF foo_2 does not exist" ?

Tommi Virtanen wrote:

Isn't the idempotency in that case "clone foo_head -> foo_2 IFF foo_2 does not exist" ?

That's almost enough for clone() (if we add O_EXCL and whitelist EEXIST for non-btrfs). It wouldn't catch something like

1 clone A->B
 2 modify A
   ...
 3 delete B
 4 <crash>
   <replay from 1>

That trick also wouldn't work for clone_range(), which doesn't create a file.

It may be that we need to make transactions idempotent at a higher level, but it'd be dependent on what you clone to, and whether it is ever modified/removed... it'd be dependent on the particular, though, and hard to analyze/verify.

FWIW even if we know what not to replay, we could still be screwed with ext4 (which does not commit everything in order):

clone A->B
 modify A
 <fs commits A (before B)>
 <crash>

On replay, we don't actually have the old A to clone to B. :(

I think the simplest solution would be:

- for all operations, set an xattr with the last op_seq to write to that file.
 - for any operation that is potentially non-idempotent, fsync(2) after doing it.
 - on replay, verify the xattr isn't == or newer to avoid re-doing the operation.

Those operations would be:

- create collection
 - clone
 - clone range

We need to set the attr on all operations to avoid something like

1 truncate B
 2 clone A->B
 3 modify A
 <crash>
 <replay 1>
 <skip 2 due to xattr>
 <replay 3>

Update: the current first pass plan is to initiate a FileStore sync after any non-idempotent operation. This updates commit_op_seq on disk and ensures that it won't be replayed.

It's also heavyweight as it calls sync(2). So it's a big hammer, but at least it's correct.

We can set a non-idempotent bool in the do_transaction method on CLONE or anything similar and then do the commit at the end (before any other operations occur under the
current OpSequencer).