Ceph

Possibly after being killed at some point, the leader ignored earlier versions when it trimmed its state, such that this happened:

$ ../../test_store_tool mon.c/store.db list paxos
paxos:1115
paxos:1116
paxos:1117
paxos:1118
paxos:2185
paxos:2186
paxos:2187
paxos:2188
paxos:2189
paxos:2190
paxos:2191
paxos:2192
paxos:2193
paxos:2194
paxos:2195
paxos:2196
paxos:2197
paxos:accepted_pn
paxos:first_committed
paxos:last_committed

Note the jump in available versions from paxos:1118 to paxos:2185 -- the missing versions were trimmed. As trim goes from 'first_committed' to 'last_committed - N', it's fair to assume that at some point the 'first_committed' version was updated without cleaning up older versions. This might have happened for a myriad of reasons, but the most likely would be some issue on the synchronization code that would update the store without getting rid of old versions.

Btw,

$ ../../test_store_tool mon.c/store.db get paxos first_committed
(paxos, first_committed)
0000 : 89 08 00 00 00 00 00 00                         : ........

$ ../../test_store_tool mon.c/store.db get paxos last_committed
(paxos, last_committed)
0000 : 95 08 00 00 00 00 00 00                         : ........

$ python -c 'print "fc {f} lc {l}".format(f=0x0889,l=0x0895)'
fc 2185 lc 2197