Ceph

it is reported that there is inconsistency between mon store leveldbs in the same cluster.

4.0G /var/lib/ceph/mon
219M /var/lib/ceph/mon
5.3G /var/lib/ceph/mon
5.3G /var/lib/ceph/mon
5.3G /var/lib/ceph/mon

this issue was originally spotted in a firefly cluster, but the symptom persists in hammer. when time elapses the size keep increasing:

5.6G /var/lib/ceph/mon
2.6G /var/lib/ceph/mon
7.7G /var/lib/ceph/mon
7.7G /var/lib/ceph/mon
7.7G /var/lib/ceph/mon

after compaction using "ceph tell mon.* compact" or ceph-monstore-tool, the larger ones shrank to 260MB.

we found leveldb 1.12 has a relevant bug https://groups.google.com/forum/#!msg/leveldb/yL6h1mAOc20/vLU64RylIdMJ. but we have an option "mon_compact_on_trim" for compacting the trimmed epochs. and it's "true" by default. so the space used by the trimmed epochs should always be compacted and reclaimed even without the bug fix from leveldb.

in other words, we do call:
CompactRange(prefix, trimmed_incremental_begin-1, trimmed_incremental_end)

but seems not all trimmed range is compacted. assuming prefix is "osdmap", begin is "123", end is "373", the range being compacted is like:

osdmap\0123..osdmap\0373

which might not cover the range of

osdmap\0full_123..osdmap\0full_373

and as https://groups.google.com/forum/#!msg/leveldb/yL6h1mAOc20/vLU64RylIdMJ put, the key range no longer touched will not be reclaimed.

so i guess we need to compact the full epochs also.