Ceph » rbd

Currently, MirrorSnapshotScheduleHandler thread gets wedged and as a result mirror snapshot scheduling is halted until ceph-mgr daemon is restarted or failed over.

The MirrorSnapshotScheduleHandler thread gets wedged when its RADOS client gets blocklisted. This would also happen to TaskHandler, PerfHandler and TrashPurgeScheduleHandler threads since they share the rbd_support module's RADOS client. To fix this issue, upon the client getting blocklisted, the handlers and the client connection would need to be shutdown, and handlers threads using a new RADOS client connection would need to be started.

The feature https://tracker.ceph.com/issues/58691, stores names of modules along with their client's address in the MgrMap. It helps improve the general debuggability of the mgr service. It will also be used in automated tests to easily identify the rbd_support module's client to blocklist and check for the module's recovery.

When a mgr is failed over using `mgr fail` command, the MgrMonitor proposes OSDMap changes where the registered clients of the mgr being failed are added to the blocklist, and proposes MgrMap changes where a standby mgr is set as active replacing the mgr being failed. The mgr being failed may not see the new MgrMap updates where it's no longer set as active. However, its modules (e.g., rbd_support) may see their clients blocklisted, and recover by registering new clients. Meanwhile, the standby mgr may see the new MgrMap before the mgr being failed does so. Since the standby mgr sees itself as newly active, it starts loading its modules and registering clients. Now, there would be mgr module clients belonging to two different mgrs trying to modify the same resource. This situation would be resolved only when the mgr being failed sees the new MgrMap where it's no longer active and kills itself. To prevent the racing of clients of two mgrs, the plan is to block the registration of the client of the mgr being failed during recovery until the client address shows up in a new MgrMap. This would force the failed Mgr to wait for the new MgrMap where it's no longer active. The work to block the registration of a mgr module's client until the client's address shows up in the MgrMap is tracked here, https://tracker.ceph.com/issues/58924

Recent work tracked by https://tracker.ceph.com/issues/58923, batches the MgrMonitor's proposal of OSDMap and MgrMap updates when dropping the mgr being failed. The batching helps reduce the delay between the blocklisting of the mgr's clients due to the OSDMap updates, and the mgr killing itself on seeing the MgrMap updates where its no longer set as active.