Ceph » RADOS

Example of the problem:

Roles:

smithi001: mon.a
smithi002: mon.b
smithi070: mon.c
smithi100 : mon.d
smithi200: mon.e

Action: Delete mon.d

smithi001: mon.a
smithi002: mon.b
smithi070: mon.c
smithi100 :
smithi200: mon.e

Monitor side:
rank is removed. A chosen monitor (it’s somewhat random) will handle the command “mon metadata e” which is requested by an mgr daemon.

Monmap->ranks are always updated before mon_metadata … so at this point in the momap perspective mon.e has rank 3 while, in the monitor perspective mon_metadata has mon.e has rank 4. The function “get_mon_metadata(rank)” in montor.cc will handle the request from the command: “mon metadata e” and will send mon.d’s metadata to the mgr instead of mon.e (the correct monitor), this is because the function relies on the variable mon_metadata.

Mgr side:
once mon.e has the incorrect metadata (that of mon.d) then when demon_state performs the insert and delete
the mechanism which will eventually insert mon.e to smithi100, even though smithi200 already has mon.e.

Result:
smithi001: mon.a
smithi002: mon.b
smithi070: mon.c
smithi100 : mon.e
smithi200: mon.e

Action: Delete mon.e

smithi001: mon.a
smithi002: mon.b
smithi070: mon.c
smithi100 :
smithi200: mon.e

Bham! Stray Daemon on the cephadm, since mon.e is still there in the list of devices.