Project

General

Profile

Actions

Bug #48933

closed

cephadm: EOFError: couldnt load message header, expected 9 bytes, got 0

Added by Gunther Heinrich over 3 years ago. Updated over 2 years ago.

Status:
Can't reproduce
Priority:
Low
Assignee:
-
Category:
cephadm
Target version:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Found several uncatched exceptions, if I remember correctly, one host was rebooting or updating at that time.

debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02] Traceback (most recent call last):
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] : Traceback (most recent call last):
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]   File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 432, in from_io
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :   File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 432, in from_io
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]     header = io.read(9)  # type 1, channel 4, payload 4
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :     header = io.read(9)  # type 1, channel 4, payload 4
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]   File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 402, in read
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :   File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 402, in read
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]     raise EOFError("expected %d bytes, got %d" % (numbytes, len(buf)))
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :     raise EOFError("expected %d bytes, got %d" % (numbytes, len(buf)))
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02] EOFError: expected 9 bytes, got 0
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] : EOFError: expected 9 bytes, got 0
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02] During handling of the above exception, another exception occurred:
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] : During handling of the above exception, another exception occurred:
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02] Traceback (most recent call last):
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] : Traceback (most recent call last):
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]   File "/lib/python3.6/site-packages/remoto/process.py", line 188, in check
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :   File "/lib/python3.6/site-packages/remoto/process.py", line 188, in check
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]     response = result.receive(timeout)
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :     response = result.receive(timeout)
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]   File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 749, in receive
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :   File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 749, in receive
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]     raise self._getremoteerror() or EOFError()
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :     raise self._getremoteerror() or EOFError()
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]   File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 967, in _thread_receiver
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :   File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 967, in _thread_receiver
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]     msg = Message.from_io(io)
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :     msg = Message.from_io(io)
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]   File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 437, in from_io
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :   File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 437, in from_io
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]     raise EOFError("couldnt load message header, " + e.args[0])
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :     raise EOFError("couldnt load message header, " + e.args[0])
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02] EOFError: couldnt load message header, expected 9 bytes, got 0
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] : EOFError: couldnt load message header, expected 9 bytes, got 0
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700  0 [cephadm ERROR root@iz-ceph-v1-osd-02]
debug 2021-01-18T13:06:36.862+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] :
debug 2021-01-18T13:06:36.938+0000 7f6e0fcd9700  0 [cephadm ERROR root] Failed to execute command: /usr/bin/python3 -u
Traceback (most recent call last):
  File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 432, in from_io
    header = io.read(9)  # type 1, channel 4, payload 4
  File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 402, in read
    raise EOFError("expected %d bytes, got %d" % (numbytes, len(buf)))
EOFError: expected 9 bytes, got 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lib/python3.6/site-packages/remoto/process.py", line 188, in check
    response = result.receive(timeout)
  File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 749, in receive
    raise self._getremoteerror() or EOFError()
  File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 967, in _thread_receiver
    msg = Message.from_io(io)
  File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 437, in from_io
    raise EOFError("couldnt load message header, " + e.args[0])
EOFError: couldnt load message header, expected 9 bytes, got 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1012, in _remote_connection
    yield (conn, connr)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1128, in _run_cephadm
    stdin=script.encode('utf-8'))
  File "/lib/python3.6/site-packages/remoto/process.py", line 209, in check
    'Failed to execute command: %s' % ' '.join(command)
RuntimeError: Failed to execute command: /usr/bin/python3 -u
debug 2021-01-18T13:06:36.938+0000 7f6e0fcd9700 -1 log_channel(cephadm) log [ERR] : Failed to execute command: /usr/bin/python3 -u
Traceback (most recent call last):
  File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 432, in from_io
    header = io.read(9)  # type 1, channel 4, payload 4
  File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 402, in read
    raise EOFError("expected %d bytes, got %d" % (numbytes, len(buf)))
EOFError: expected 9 bytes, got 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lib/python3.6/site-packages/remoto/process.py", line 188, in check
    response = result.receive(timeout)
  File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 749, in receive
    raise self._getremoteerror() or EOFError()
  File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 967, in _thread_receiver
    msg = Message.from_io(io)
  File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 437, in from_io
    raise EOFError("couldnt load message header, " + e.args[0])
EOFError: couldnt load message header, expected 9 bytes, got 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1012, in _remote_connection
    yield (conn, connr)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 1128, in _run_cephadm
    stdin=script.encode('utf-8'))
  File "/lib/python3.6/site-packages/remoto/process.py", line 209, in check
    'Failed to execute command: %s' % ' '.join(command)
RuntimeError: Failed to execute command: /usr/bin/python3 -u

Actions

Also available in: Atom PDF