pgmap updates from OSDMap can be delayed indefinitely
We saw a customer cluster in which a full OSD had been removed from the OSDMap, but after almost two hours that change had not propagated to the pgmap's list of full OSDs. Going through monitor logs, every time the PGMonitor tried to run update_from_osdmap, either the osdmap was unreadable or the pgmap was unwriteable.
After discussion with Joao and Sage, we think it's safe in our current implementation to simply drop the is_readable() check on the osdmonitor in this case, because while we might see an out-of-date map, we won't see an invalid one. In future we'll need to always provide a stable readable map for situations like this.
#2 Updated by Greg Farnum almost 5 years ago
I should also note that I suspect this condition might have been exacerbated by our full map handling. We probably had every daemon/client in the cluster trying to get map updates from the monitor (increasing load), and maybe the OSDs would have been reporting more frequently? Because the updates seemed to be happening just fine before the full flag got set.