Actions
Bug #4185
closedPython multiprocessing exhibiting odd behaviour with librados
% Done:
0%
Source:
Community (user)
Tags:
python
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
The following code snippet yields unexpected results:
#!/usr/bin/python import multiprocessing import rbd import rados import logging # some multiprocessing usefol logging mpl = multiprocessing.log_to_stderr() mpl.setLevel(logging.INFO) def childconnect(): # set up a rados object threadcluster = rados.Rados(conffile='/etc/ceph/ceph.conf') # and connect. but this fails, mon logs show 'unexpected key: req.key=0' threadcluster.connect() # :( threadcluster.shutdown() if __name__ == '__main__': # set up a main connection cluster = rados.Rados(conffile='/etc/ceph/ceph.conf') cluster.connect() # do some generic stuff first, before we set up child processes to do more work. job = [] job.append(multiprocessing.Process(target=childconnect)) job.append(multiprocessing.Process(target=childconnect)) for j in job: j.start() j.join() # and we're done. cluster.shutdown()
If the cluster object and cluster.connect/shutdown is removed from the main process, child connections work fine. If kept as is, child connections consistently fail:
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "./mpcephfail.py", line 14, in childconnect
threadcluster.connect()
File "/usr/lib/python2.7/dist-packages/rados.py", line 185, in connect
raise make_ex(ret, "error calling connect")
PermissionError: error calling connect
mons log the following error:
2013-02-18 10:52:54.592627 7f871c6f1700 0 cephx server client.admin: unexpected key: req.key=0 expected_key=e0c36589ad8a04e7
Can be worked around by doing everything CEPH related in a child process and not connecting in the main process whatsoever, but it seems to be an issue with the python bindings.
Updated by Samuel Just over 10 years ago
- Status changed from New to Won't Fix
Multiprocessing forks, and that's a problem.
Updated by Evan Felix about 10 years ago
Some notes for the Next guy that comes across this issue:
You can use the multiprocessing managers to push all the rados calls to a single process. The code cnage be changed to this:
#!/usr/bin/python
import multiprocessing
import multiprocessing.managers
import rbd
import rados
import logging
# some multiprocessing usefol logging
mpl = multiprocessing.log_to_stderr()
mpl.setLevel(logging.INFO)
class MyManager(multiprocessing.managers.BaseManager):
pass
MyManager.register("Rados",rados.Rados)
def childconnect(mgr):
# set up a rados object
threadcluster = mgr.Rados(conffile='/etc/ceph/ceph.conf')
# and connect. but this fails, mon logs show 'unexpected key: req.key=0'
threadcluster.connect()
# :(
threadcluster.shutdown()
if __name__ == '__main__':
mgr = MyManager()
mgr.start()
# set up a main connection
cluster = mgr.Rados(conffile='/etc/ceph/ceph.conf')
cluster.connect()
# do some generic stuff first, before we set up child processes to do more work.
job = []
job.append(multiprocessing.Process(target=childconnect,args=(mgr,)))
job.append(multiprocessing.Process(target=childconnect,args=(mgr,)))
for j in job:
j.start()
j.join()
# and we're done.
cluster.shutdown()
Actions