osd: unhealthy osd cannot be marked down in time
Before an unhealthy osd is marked down by mon, other osd may choose it as
heartbeat peer and then report an incorrect failure time(first_tx) to mon.
Shutdown cluster_network and public_network of an osd node several times.
#2 Updated by wencong wan 4 months ago
Radoslaw Zarzynski wrote:
Could you please clarify a bit? Do you mean there some extra, unnecessary (from the POV of jugging whether an OSD is down or not) messages that just update the markdown timestamp?
Whether an OSD is down or not is determined by mon.If either of the following two conditions is met, mon will mark an osd as down.
1、Mon does not receive osd_beacon message of an osd for more than 900s(mon_osd_report_timeout)
2、Mon receive failure_report message from 2(mon_osd_min_down_reporters) osds on different host(mon_osd_reporter_subtree_level) and the fault lasted for a period of time（now - fi.get_failed_since() > grace）.
get_failed_since return the max failed time of all reporters. if some osd choose the unhealthy osd as heartbeat peer,they will never receive heartbeat reply from the unhealthy osd. So these osds will report the first time of sending heartbeat as the failure time of the unhealthy osd. The condition "now - fi.get_failed_since() > grace" cannot be met.