Ceph » rbd

How to reproduce:

1. Create image, connect to VM and install any OS. Ensure that KVM uses discard feature and virtio-scsi driver.
2. Run `fstrim --all` in guest OS and ensure that disk usage in reduced in Ceph (i.e. through `rbd du` command)
3. Create a snapshot of RBD image.
4. Remove some files inside guest image
5. Run `fstrim --all` again.
6. See that amount of used space is actually increases for that image!

I suggest, that this happen due to logic that writes (allocates) zeroes instead of unmapping regions, while discarding regions on images, that have snapshot.

There are three solutions for that (we should choose one of them):

1. Just decrement reference count on that region, unmap this region form RBD image. Exactly as it happens for image without snapshots. I don't understand why this was not already done. (Preferred)
2. Just remove any records about that region, so it will refer to original region of base image (if it was allocated before snapshotting). In that case, reading from discarded region will return some old data. It is allowed for SSD for example. (Very simple, but not friendly for some usage, see comments below)
3. Introduce `whiteout` flag in RBD metadata for case when region [that was allocated in base image] is discarded.

AFAIK, there is flag for SCSI device specifying if discarded regions will return zeroes on reading.

This happens on Kraken (server) and jewel on client.