Bug #46990
Updated by Sebastian Wagner over 3 years ago
<pre>
[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Failed to execute command: /usr/bin/python3 -u
Module 'cephadm' has failed: Failed to execute command: /usr/bin/python3 -u
</pre>
<pre>
Aug 17 13:42:20 master bash[19272]: Traceback (most recent call last):
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/execnet/gateway_base.py", line 432, in from_io
Aug 17 13:42:20 master bash[19272]: header = io.read(9) # type 1, channel 4, payload 4
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/execnet/gateway_base.py", line 402, in read
Aug 17 13:42:20 master bash[19272]: raise EOFError("expected %d bytes, got %d" % (numbytes, len(buf)))
Aug 17 13:42:20 master bash[19272]: EOFError: expected 9 bytes, got 0
Aug 17 13:42:20 master bash[19272]: During handling of the above exception, another exception occurred:
Aug 17 13:42:20 master bash[19272]: Traceback (most recent call last):
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/remoto/process.py", line 188, in check
Aug 17 13:42:20 master bash[19272]: response = result.receive(timeout)
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/execnet/gateway_base.py", line 749, in receive
Aug 17 13:42:20 master bash[19272]: raise self._getremoteerror() or EOFError()
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/execnet/gateway_base.py", line 967, in _thread_receiver
Aug 17 13:42:20 master bash[19272]: msg = Message.from_io(io)
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/execnet/gateway_base.py", line 437, in from_io
Aug 17 13:42:20 master bash[19272]: raise EOFError("couldnt load message header, " + e.args[0])
Aug 17 13:42:20 master bash[19272]: EOFError: couldnt load message header, expected 9 bytes, got 0
Aug 17 13:42:20 master bash[19272]: During handling of the above exception, another exception occurred:
Aug 17 13:42:20 master bash[19272]: Traceback (most recent call last):
Aug 17 13:42:20 master bash[19272]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1035, in _remote_connection
Aug 17 13:42:20 master bash[19272]: yield (conn, connr)
Aug 17 13:42:20 master bash[19272]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1131, in _run_cephadm
Aug 17 13:42:20 master bash[19272]: stdin=script.encode('utf-8'))
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/remoto/process.py", line 209, in check
Aug 17 13:42:20 master bash[19272]: 'Failed to execute command: %s' % ' '.join(command)
Aug 17 13:42:20 master bash[19272]: RuntimeError: Failed to execute command: /usr/bin/python3 -u
Aug 17 13:42:20 master bash[19272]: debug 2020-08-17T11:42:20.900+0000 7f2bbda3c700 -1 log_channel(cephadm) log [ERR] : Failed to execute command: /usr/bin/python3 -u
Aug 17 13:42:20 master bash[19272]: Traceback (most recent call last):
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/execnet/gateway_base.py", line 432, in from_io
Aug 17 13:42:20 master bash[19272]: header = io.read(9) # type 1, channel 4, payload 4
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/execnet/gateway_base.py", line 402, in read
Aug 17 13:42:20 master bash[19272]: raise EOFError("expected %d bytes, got %d" % (numbytes, len(buf)))
Aug 17 13:42:20 master bash[19272]: EOFError: expected 9 bytes, got 0
Aug 17 13:42:20 master bash[19272]: During handling of the above exception, another exception occurred:
Aug 17 13:42:20 master bash[19272]: Traceback (most recent call last):
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/remoto/process.py", line 188, in check
Aug 17 13:42:20 master bash[19272]: response = result.receive(timeout)
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/execnet/gateway_base.py", line 749, in receive
Aug 17 13:42:20 master bash[19272]: raise self._getremoteerror() or EOFError()
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/execnet/gateway_base.py", line 967, in _thread_receiver
Aug 17 13:42:20 master bash[19272]: msg = Message.from_io(io)
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/execnet/gateway_base.py", line 437, in from_io
Aug 17 13:42:20 master bash[19272]: raise EOFError("couldnt load message header, " + e.args[0])
Aug 17 13:42:20 master bash[19272]: EOFError: couldnt load message header, expected 9 bytes, got 0
Aug 17 13:42:20 master bash[19272]: During handling of the above exception, another exception occurred:
Aug 17 13:42:20 master bash[19272]: Traceback (most recent call last):
Aug 17 13:42:20 master bash[19272]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1035, in _remote_connection
Aug 17 13:42:20 master bash[19272]: yield (conn, connr)
Aug 17 13:42:20 master bash[19272]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1131, in _run_cephadm
Aug 17 13:42:20 master bash[19272]: stdin=script.encode('utf-8'))
Aug 17 13:42:20 master bash[19272]: File "/usr/lib/python3.6/site-packages/remoto/process.py", line 209, in check
Aug 17 13:42:20 master bash[19272]: 'Failed to execute command: %s' % ' '.join(command)
Aug 17 13:42:20 master bash[19272]: RuntimeError: Failed to execute command: /usr/bin/python3 -u
Aug 17 13:42:21 master bash[19272]: Warning: Permanently added 'master' (ECDSA) to the list of known hosts.
</pre>
execnet is again super helpful.
Fortunately, we were able to recover from this, as we're calling _reset_con() in that case.