Ceph

This might also be the cause of the very small number of leaked messages I think Sam or Joao mentioned to me.

That said, these outdated beacons ought to be rejected based on the seq and info.state_seq values, but it seems they aren't always, as in the referenced ticket.

after the old MDS died, a new stand-by MDS replaced it. so MDSMonitor erased all the history (e.g. pending_mdsmap.mds_info, last_beacon) related to the old gid when it did the housekeeping. that's why after 4123 replaced the 4111. MDSMonitor forgot everything about 4111, including the last msg seq# from it. and the peon monitor kept resending the mdsbeacon message (mdsbeacon(4111/a-s up:boot seq 1 v0)) from the dead MDS tirelessly. this outdated message misled the lead mon and eventually brought down the innocent new MDS.

D'oh, well spotted. We could maybe check for matching entity_inst_t when handling beacons, but that gets complicated with some of our naming and takeover logic.

For now just replying to beacons and stopping the retransmission is fine.

see the discussions on https://github.com/ceph/ceph/pull/4702

per greg,

So we need some kind of state message to tell peons to drop forwarded messages because they've been received and aren't getting a response.

so we can not reply to mdsbeacons ignored by the leader.