Ceph

It seems that Ceph caching doesn't take removed ( whitened out) objects
into accounts when checks for the need to flush the cache.

Following pools have been created:
./ceph -c ceph.conf osd pool create cachepool 12 12
./ceph -c ceph.conf osd pool create ecpool 12 12 erasure
./ceph -c ceph.conf osd tier add ecpool cachepool
./ceph -c ceph.conf osd tier cache-mode cachepool writeback

./ceph -c ceph.conf osd tier set-overlay ecpool cachepool
./ceph -c ceph.conf osd pool set cachepool hit_set_type bloom
./ceph -c ceph.conf osd pool set cachepool target_max_bytes 1000000

Then doing the following in a loop:
- Write 16K data to a new object with unique name
- Remove object.

causes cache pool object count to grow permanently:
_{ceph/ceph_com/src# ./rados -c ceph.conf df
pool name KB objects clones degraded unfound rd rd KB wr wr KB
cachepool 48 285 0 0 0 0 0 567 4560
...
ceph/ceph_com/src# ./rados -c ceph.conf df
pool name KB objects clones degraded unfound rd rd KB wr wr KB
cachepool 0 5947 0 0 0 0 0 11894 95152
...
etc.}

The same applies to disk usage reported by du command:
_{/ceph/ceph_com/src# du -h dev -s
461M dev
...
/ceph/ceph_com/src# du -h dev -s
465M dev}

From code analysis it looks like following two parameters affect cache
flush triggering: target_max_bytes and target_max_objects.
When the latter set to 0 (by default) cache flush wouldn't happen no
matter how many removed objects are in the cache since their size is
supposed to be 0 bytes. But in fact that's not true - empty
files(objects) consume some space too. Thus potentially one can even
completely overfill the cache with removed objects.