CDM 06-MAR-2024¶

- [Brad] Initial discussions around protecting the manager -> mon queue from overload by rogue modules

• Get a better understanding of what this throttle ("throttle-mgr_mon_messsages") represents (and why it seems to have any relation to the volume plugin)
• Try a test where the volumes plugin is artificially hung
  • Fail the fs (cephfs fail [fsname]), spawn a bunch of clients that send messages to the volumes plugin and see if we see this scenario happening. Something like `ceph fs subvolume create <fs_name> subvol.XYZ`.
• Etherpad to coordinate the response: https://pad.ceph.com/p/throttle-mgr_mon_messsages_engages_heavily

- [Patrick/Venky; if we're available] ceph-mgr throttles hit after module (volumes) finisher thread queue fills up with commands. This causes messenger threads to throttle commands/queries from other clients. Idea for discussion: throttle clients by connection to ensure fairness.

• Allocate each client a number of slots in the throttle and use this to ensure some level of fairness in servicing each module
• Note that we may have a far better understanding of how to resolve this once we have a far better understanding of the issue described in item 1 which is hitherto not really comprehensively understood.