Ceph

Add a debug option to probalistically (or every 5th time?) drop the message and force the monclient to reconnect.

Observed this preventing an osd from booting (it was stuck asking for maps).

if an OSD is booting but the monclient session got reset, and the get_osdmap message was either dropped or its reply got dropped. the mon client should try to reopen_session() and re-authorize itself. and OSD::ms_handle_connect() will get called when the monc reconnects to the monitor. in which, OSD will start_boot() since it is_booting(). and in start_boot(), OSD will send another query to monitor for osdmaps. so the osdmap will be received anyway.

so i think i must be missing something here, Sam and Sage, could you guys shed some light on it? thanks!

Kefu Chai wrote:

Observed this preventing an osd from booting (it was stuck asking for maps).

if an OSD is booting but the monclient session got reset, and the get_osdmap message was either dropped or its reply got dropped. the mon client should try to reopen_session() and re-authorize itself. and OSD::ms_handle_connect() will get called when the monc reconnects to the monitor. in which, OSD will start_boot() since it is_booting(). and in start_boot(), OSD will send another query to monitor for osdmaps. so the osdmap will be received anyway.

... except that osd_epoch cache on the mon will still say that the osd already got the OSDMap (that it didn't), so the subscribe will be ignored. See check_subs() -> send_incremental().

I think the fix is to destroy the osd_epoch map<> and move that information into the MonSession, so that when the connection is lost the state gets cleaned up.

thanks sage, i got it now =)