pybind/mgr/volumes: investigate moving calls which may block on libcephfs into another thread
To not block the ceph-mgr finisher thread on any calls out to cephfs. This can have disastrous consequences as the mgr will then stop functioning.
(I think it may need changes to ceph-mgr to support this. Namely, the core ceph-mgr code handles the return from the command and creates a reply message. We need that reply message to be constructed within the module and sent out.)
#2 Updated by Vikhyat Umrao 4 months ago
Downstream BZ - https://bugzilla.redhat.com/show_bug.cgi?id=2114615
#4 Updated by Venky Shankar 3 months ago
Spoke to Kotersh today - we may want to introduce an async command execution interface in plugins that the finisher thread would call and "handover" the request to be replied back by the plugin itself. Plugins can choose to implement this execution "mode" or use the default blocking call by the finisher thread.
#5 Updated by Kotresh Hiremath Ravishankar 3 months ago
- Pull request ID set to 47893
Discussion Summary with Patrick
1. Have a thread for each module to execute module commands. Since the finisher thread infrastructure is already in place, it's better to use one finisher thread per module.
Currently with the draft PR, there is one finisher thread per module and one generic finisher thread via which all other things like config, notify is done. This is different from the
comment 4. Both has it's pros and cons. With this approach, if any command is stuck in a python module, only that module is affected (the subsequent module commands waits) and other
module commands goes through. This is comparatively easy to implement. The comment 4's approach needs change in every python module and the effect of asynchronous nature is to be tested
as all module commands is asynchronous.
2. Add a warning if the finisher thread's queue is growing or if the it takes more than 15 secs for the single command.
3. Add extensive performance counters for osdmap, mdsmap changes, command processed (op time, command count)