Ceph

EC overwrites preclude our current checksumming strategy. Instead, on each shard, we'll maintain a set of omap entries (prefixed so that we can use the omap space on the object for other things later) where each entry contains N per-stripe checksums. N should be configurable at pool creation time or when the pool is switched to allow overwrites (and is fixed forever after). Notably, we need to be able to switch an append-only ec pool to allow overwrites.

When an object which was written in append-only mode is overwritten, the OSD should clear the append-only checksum attr, set the checksum for the modified extents, and initialize all of the other block checksums to a value which indicates that it is non-determinant. When an append-only object is deep-scrubbed after the mode is switched, the correct checksums should be immediately written out.

We also need a cache for unstable checksums.

As with several of the other features, this feature is complete once when the ec overwrites testing flag is enabled, current ec pools use the new checksumming scheme.