Ceph » mgr

I think the original intent was only to pass data around between modules, not things that could be executed (see the commit message in https://github.com/ceph/ceph/pull/22951/commits/f02316adb4baf4dceaab79cc0ef4c2acdb544f3e)

Tim Serong wrote:

I think the original intent was only to pass data around between modules

yeah, you don't even need to execute code. passing objects is enough to get into the realm of funny things

Is that just for objects that contain code (i.e. methods), or objects that only contain data? In any case, I think we probably just can't pass objects around between modules, and need to stick to data only.

We can't fix this by removing subinterpreters and going back to a single interpreter that runs all mgr modules, because that will mean that various mgr module dependencies conflict with each other: think two mgr modules that both use cherrypy -- if these are run in a single interpreter, only one will work.

If we do remove subinterepters in future, I believe we will have to switch to an even more isolated structure (e.g. a separate process per module with some form of IPC (or shared memory? not sure)), which also wouldn't support passing objects around.

Tim Serong wrote:

Is that just for objects that contain code (i.e. methods), or objects that only contain data? In any case, I think we probably just can't pass objects around between modules, and need to stick to data only.

there are actually at least two parts of this problem:

1. Passing user defined types around is enough to cause problems with module identity. Anything that depends on module identity breaks: try: except, isinstance, issubclass, etc...
2. Executing code outside of it's sub-interpreter context, e.g. all code called by remote, like all methods in deepsea.module.DeepSeaOrchestrator that implement methods of Prchestrator including wait: They run in that infamous "Reduced Execution Environment"

We can't fix this by removing subinterpreters and going back to a single interpreter that runs all mgr modules, because that will mean that various mgr module dependencies conflict with each other: think two mgr modules that both use cherrypy -- if these are run in a single interpreter, only one will work.

I've had some bad experience in having multiple sub-interpreters running HTTPS webservers using a common OpenSSL library. I don't know if we really want to have so many different web servers executing within the mgr.

If we do remove subinterepters in future, I believe we will have to switch to an even more isolated structure (e.g. a separate process per module with some form of IPC (or shared memory? not sure)), which also wouldn't support passing objects around.

Maybe that's already enough? Like don't try to pass any Python objects around and don't directly call any functions and instead perform some RPC between sub-interpreters.

Sebastian Wagner wrote:

We can't fix this by removing subinterpreters and going back to a single interpreter that runs all mgr modules, because that will mean that various mgr module dependencies conflict with each other: think two mgr modules that both use cherrypy -- if these are run in a single interpreter, only one will work.

I've had some bad experience in having multiple sub-interpreters running HTTPS webservers using a common OpenSSL library. I don't know if we really want to have so many different web servers executing within the mgr.

cherrypy was just an example - the general problem is that if two mgr modules have some python module as a dependency which in turn has some global state, it's going to break.

If we do remove subinterepters in future, I believe we will have to switch to an even more isolated structure (e.g. a separate process per module with some form of IPC (or shared memory? not sure)), which also wouldn't support passing objects around.

Maybe that's already enough? Like don't try to pass any Python objects around and don't directly call any functions and instead perform some RPC between sub-interpreters.

If we're going to develop an RPC solution, I'd suggest waiting until we get rid of subinterpreters and switch to multiple processes. In the meantime, I don't see the benefit of developing some sort of RPC system to use with subinterpreters, over, say, just passing simple data types around.

I just don't know if multiple processes are the best way. One big advantage by using one mgr process is the ability to have shared mgr data structures in the mgr memory that we don't need to replicate and synchronize between modules.

#35985 and #38467 are about thread deadlocks. What evidence do you have that they are related to the issue reported here?

Sebastian Wagner wrote:

I just don't know if multiple processes are the best way. One big advantage by using one mgr process is the ability to have shared mgr data structures in the mgr memory that we don't need to replicate and synchronize between modules.

I agree completely. But I suspect that eventually we'll have to ditch subinterpreters and split into separate processes, because subinterpreters are a relatively underused/undertested/underdocumented feature of python, and I don't want to have to do more weird things like https://github.com/ceph/ceph/pull/25585 to keep everything working :-)

I don't want to have to do more weird things like https://github.com/ceph/ceph/pull/25585 to keep everything working :-)

And I also completely agree wit this :-)

Brad Hubbard wrote:

#35985 and #38467 are about thread deadlocks. What evidence do you have that they are related to the issue reported here?

I don't think they are related.

Sebastian Wagner wrote:

Brad Hubbard wrote:

#35985 and #38467 are about thread deadlocks. What evidence do you have that they are related to the issue reported here?

I don't think they are related.

But you attached them as "Related issues" ?

removed unrelated issues.

may relate to https://lwn.net/Articles/754162/