Project

General

Profile

Bug #53358

mgr/cephadm: ssh errors too verbose and timeout too long when can't connect to host

Added by Adam King about 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When hosts are found to be offline a health warning will be raised with a corresponding error message. However, the current error messages are far too verbose and contain mostly non helpful information.

NOTE: seemingly an easy way to recreate this is to add a nonexistent host e.g. "ceph orch host add fake-host 2.2.2.2" will sit there for a few minutes then spit out one of these long messages

An excerpt from a "ceph health detail" call

[WRN] CEPHADM_HOST_CHECK_FAILED: 2 hosts fail cephadm check
    host storage-01-08014 (10.221.1.56) failed check: Failed to connect to storage-01-08014 (10.221.1.56). Connection lost
Log: Opening SSH connection to 10.221.1.56, port 22
Opening SSH connection to 10.221.1.111, port 22
[conn=1687] Connected to SSH server at 10.221.1.56, port 22
[conn=1687]   Local address: 10.221.1.188, port 42830
[conn=1687]   Peer address: 10.221.1.56, port 22
[conn=1687] Sending version SSH-2.0-AsyncSSH_2.7.0
[conn=1688] Connected to SSH server at 10.221.1.111, port 22
[conn=1688]   Local address: 10.221.1.188, port 45254
[conn=1688]   Peer address: 10.221.1.111, port 22
[conn=1688] Sending version SSH-2.0-AsyncSSH_2.7.0
[conn=1687] Connection lost
[conn=88, chan=132] Set write buffer limits: low-water=16384, high-water=65536
[conn=88, chan=132] Requesting new SSH session
[conn=88, chan=132]   Initial recv window 2097152, packet size 32768
[conn=88, pktid=1191] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=88, chan=132, pktid=1192] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 84  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=84, chan=132] Set write buffer limits: low-water=16384, high-water=65536
[conn=84, chan=132] Requesting new SSH session
[conn=84, chan=132]   Initial recv window 2097152, packet size 32768
[conn=84, pktid=1193] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=84, chan=132, pktid=1194] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 84  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=80, chan=132] Set write buffer limits: low-water=16384, high-water=65536
[conn=80, chan=132] Requesting new SSH session
[conn=80, chan=132]   Initial recv window 2097152, packet size 32768
[conn=80, pktid=1191] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=80, chan=132, pktid=1192] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 84  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=1687] Aborting connection

    host storage-02-08027 (10.221.1.111) failed check: Can't communicate with remote host `10.221.1.111`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer
Log: Opening SSH connection to 10.221.1.111, port 22
[conn=1687] Connected to SSH server at 10.221.1.56, port 22
[conn=1687]   Local address: 10.221.1.188, port 42830
[conn=1687]   Peer address: 10.221.1.56, port 22
[conn=1687] Sending version SSH-2.0-AsyncSSH_2.7.0
[conn=1688] Connected to SSH server at 10.221.1.111, port 22
[conn=1688]   Local address: 10.221.1.188, port 45254
[conn=1688]   Peer address: 10.221.1.111, port 22
[conn=1688] Sending version SSH-2.0-AsyncSSH_2.7.0
[conn=1687] Connection lost
[conn=88, chan=132] Set write buffer limits: low-water=16384, high-water=65536
[conn=88, chan=132] Requesting new SSH session
[conn=88, chan=132]   Initial recv window 2097152, packet size 32768
[conn=88, pktid=1191] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=88, chan=132, pktid=1192] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 84  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=84, chan=132] Set write buffer limits: low-water=16384, high-water=65536
[conn=84, chan=132] Requesting new SSH session
[conn=84, chan=132]   Initial recv window 2097152, packet size 32768
[conn=84, pktid=1193] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=84, chan=132, pktid=1194] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 84  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=80, chan=132] Set write buffer limits: low-water=16384, high-water=65536
[conn=80, chan=132] Requesting new SSH session
[conn=80, chan=132]   Initial recv window 2097152, packet size 32768
[conn=80, pktid=1191] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=80, chan=132, pktid=1192] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 84  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=1687] Aborting connection
[conn=88, pktid=1329] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 84 00 00 00 00 00 00 00 00 00 00 80  [...............
  00000010: 00                                               .
[conn=88, chan=132]   Initial send window 0, packet size 32768
[conn=84, pktid=1325] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 84 00 00 00 00 00 00 00 00 00 00 80  [...............
  00000010: 00                                               .
[conn=84, chan=132]   Initial send window 0, packet size 32768
[conn=80, pktid=1328] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 84 00 00 00 00 00 00 00 00 00 00 80  [...............
  00000010: 00                                               .
[conn=80, chan=132]   Initial send window 0, packet size 32768
[conn=88, chan=132]   Command: sudo which python3
[conn=88, pktid=1193] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=88, chan=132, pktid=1194] Sent MSG_CHANNEL_REQUEST (98), 36 bytes
  00000000: 62 00 00 00 00 00 00 00 04 65 78 65 63 01 00 00  b........exec...
  00000010: 00 12 73 75 64 6f 20 77 68 69 63 68 20 70 79 74  ..sudo which pyt
  00000020: 68 6f 6e 33                                      hon3
[conn=84, chan=132]   Command: sudo which python3
[conn=84, pktid=1195] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=84, chan=132, pktid=1196] Sent MSG_CHANNEL_REQUEST (98), 36 bytes
  00000000: 62 00 00 00 00 00 00 00 04 65 78 65 63 01 00 00  b........exec...
  00000010: 00 12 73 75 64 6f 20 77 68 69 63 68 20 70 79 74  ..sudo which pyt
  00000020: 68 6f 6e 33                                      hon3
[conn=80, chan=132]   Command: sudo which python3
[conn=80, pktid=1193] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=80, chan=132, pktid=1194] Sent MSG_CHANNEL_REQUEST (98), 36 bytes
  00000000: 62 00 00 00 00 00 00 00 04 65 78 65 63 01 00 00  b........exec...
  00000010: 00 12 73 75 64 6f 20 77 68 69 63 68 20 70 79 74  ..sudo which pyt
  00000020: 68 6f 6e 33                                      hon3
[conn=88, chan=132, pktid=1330] Received MSG_CHANNEL_WINDOW_ADJUST (93), 9 bytes
  00000000: 5d 00 00 00 84 00 20 00 00                       ]..... ..
[conn=88, chan=132] Received window adjust of 2097152 bytes, new window 2097152
[conn=88, chan=132, pktid=1331] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 84                                   c....
[conn=88, chan=132, pktid=1332] Received MSG_CHANNEL_DATA (94), 26 bytes
  00000000: 5e 00 00 00 84 00 00 00 11 2f 75 73 72 2f 62 69  ^......../usr/bi
  00000010: 6e 2f 70 79 74 68 6f 6e 33 0a                    n/python3.
[conn=88, chan=132] Received 17 data bytes
[conn=88, chan=132, pktid=1333] Received MSG_CHANNEL_REQUEST (98), 25 bytes
  00000000: 62 00 00 00 84 00 00 00 0b 65 78 69 74 2d 73 74  b........exit-st
  00000010: 61 74 75 73 00 00 00 00 00                       atus.....
[conn=88, chan=132] Received exit status 0
[conn=88, chan=132, pktid=1334] Received MSG_CHANNEL_EOF (96), 5 bytes
  00000000: 60 00 00 00 84                                   `....
[conn=88, chan=132] Received EOF
[conn=88, chan=132, pktid=1335] Received MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 84                                   a....
[conn=88, chan=132] Received channel close
[conn=88, pktid=1195] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=88, chan=132, pktid=1196] Sent MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 00                                   a....
[conn=84, chan=132, pktid=1326] Received MSG_CHANNEL_WINDOW_ADJUST (93), 9 bytes
  00000000: 5d 00 00 00 84 00 20 00 00                       ]..... ..
[conn=84, chan=132] Received window adjust of 2097152 bytes, new window 2097152
[conn=84, chan=132, pktid=1327] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 84                                   c....
[conn=84, chan=132, pktid=1328] Received MSG_CHANNEL_DATA (94), 26 bytes
  00000000: 5e 00 00 00 84 00 00 00 11 2f 75 73 72 2f 62 69  ^......../usr/bi
  00000010: 6e 2f 70 79 74 68 6f 6e 33 0a                    n/python3.
[conn=84, chan=132] Received 17 data bytes
[conn=84, chan=132, pktid=1329] Received MSG_CHANNEL_EOF (96), 5 bytes
  00000000: 60 00 00 00 84                                   `....
[conn=84, chan=132] Received EOF
[conn=84, chan=132, pktid=1330] Received MSG_CHANNEL_REQUEST (98), 25 bytes
  00000000: 62 00 00 00 84 00 00 00 0b 65 78 69 74 2d 73 74  b........exit-st
  00000010: 61 74 75 73 00 00 00 00 00                       atus.....
[conn=84, chan=132] Received exit status 0
[conn=84, chan=132, pktid=1331] Received MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 84                                   a....
[conn=84, chan=132] Received channel close
[conn=84, pktid=1197] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=84, chan=132, pktid=1198] Sent MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 00                                   a....
[conn=80, chan=132, pktid=1329] Received MSG_CHANNEL_WINDOW_ADJUST (93), 9 bytes
  00000000: 5d 00 00 00 84 00 20 00 00                       ]..... ..
[conn=80, chan=132] Received window adjust of 2097152 bytes, new window 2097152
[conn=80, chan=132, pktid=1330] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 84                                   c....
[conn=80, chan=132, pktid=1331] Received MSG_CHANNEL_DATA (94), 26 bytes
  00000000: 5e 00 00 00 84 00 00 00 11 2f 75 73 72 2f 62 69  ^......../usr/bi
  00000010: 6e 2f 70 79 74 68 6f 6e 33 0a                    n/python3.
[conn=80, chan=132] Received 17 data bytes
[conn=80, chan=132, pktid=1332] Received MSG_CHANNEL_EOF (96), 5 bytes
  00000000: 60 00 00 00 84                                   `....
[conn=80, chan=132] Received EOF
[conn=80, chan=132, pktid=1333] Received MSG_CHANNEL_REQUEST (98), 25 bytes
  00000000: 62 00 00 00 84 00 00 00 0b 65 78 69 74 2d 73 74  b........exit-st
  00000010: 61 74 75 73 00 00 00 00 00                       atus.....
[conn=80, chan=132] Received exit status 0
[conn=80, chan=132, pktid=1334] Received MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 84                                   a....
[conn=80, chan=132] Received channel close
[conn=80, pktid=1195] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=80, chan=132, pktid=1196] Sent MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 00                                   a....
[conn=162, chan=130] Set write buffer limits: low-water=16384, high-water=65536
[conn=162, chan=130] Requesting new SSH session
[conn=162, chan=130]   Initial recv window 2097152, packet size 32768
[conn=162, pktid=1179] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=162, chan=130, pktid=1180] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 82  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=164, chan=130] Set write buffer limits: low-water=16384, high-water=65536
[conn=164, chan=130] Requesting new SSH session
[conn=164, chan=130]   Initial recv window 2097152, packet size 32768
[conn=164, pktid=1179] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=164, chan=130, pktid=1180] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 82  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=88, chan=132] Reading from channel started
[conn=84, chan=132] Reading from channel started
[conn=80, chan=132] Reading from channel started
[conn=88, chan=132] Channel closed
[conn=84, chan=132] Channel closed
[conn=80, chan=132] Channel closed
[conn=162, pktid=1310] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 82 00 00 00 00 00 00 00 00 00 00 80  [...............
  00000010: 00                                               .
[conn=162, chan=130]   Initial send window 0, packet size 32768
[conn=164, pktid=1312] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 82 00 00 00 00 00 00 00 00 00 00 80  [...............
  00000010: 00                                               .
[conn=164, chan=130]   Initial send window 0, packet size 32768
[conn=162, chan=130]   Command: sudo which python3
[conn=162, pktid=1181] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=162, chan=130, pktid=1182] Sent MSG_CHANNEL_REQUEST (98), 36 bytes
  00000000: 62 00 00 00 00 00 00 00 04 65 78 65 63 01 00 00  b........exec...
  00000010: 00 12 73 75 64 6f 20 77 68 69 63 68 20 70 79 74  ..sudo which pyt
  00000020: 68 6f 6e 33                                      hon3
[conn=164, chan=130]   Command: sudo which python3
[conn=164, pktid=1181] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=164, chan=130, pktid=1182] Sent MSG_CHANNEL_REQUEST (98), 36 bytes
  00000000: 62 00 00 00 00 00 00 00 04 65 78 65 63 01 00 00  b........exec...
  00000010: 00 12 73 75 64 6f 20 77 68 69 63 68 20 70 79 74  ..sudo which pyt
  00000020: 68 6f 6e 33                                      hon3
[conn=162, chan=130, pktid=1311] Received MSG_CHANNEL_WINDOW_ADJUST (93), 9 bytes
  00000000: 5d 00 00 00 82 00 20 00 00                       ]..... ..
[conn=162, chan=130] Received window adjust of 2097152 bytes, new window 2097152
[conn=162, chan=130, pktid=1312] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 82                                   c....
[conn=162, chan=130, pktid=1313] Received MSG_CHANNEL_DATA (94), 26 bytes
  00000000: 5e 00 00 00 82 00 00 00 11 2f 75 73 72 2f 62 69  ^......../usr/bi
  00000010: 6e 2f 70 79 74 68 6f 6e 33 0a                    n/python3.
[conn=162, chan=130] Received 17 data bytes
[conn=162, chan=130, pktid=1314] Received MSG_CHANNEL_EOF (96), 5 bytes
  00000000: 60 00 00 00 82                                   `....
[conn=162, chan=130] Received EOF
[conn=162, chan=130, pktid=1315] Received MSG_CHANNEL_REQUEST (98), 25 bytes
  00000000: 62 00 00 00 82 00 00 00 0b 65 78 69 74 2d 73 74  b........exit-st
  00000010: 61 74 75 73 00 00 00 00 00                       atus.....
[conn=162, chan=130] Received exit status 0
[conn=162, chan=130, pktid=1316] Received MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 82                                   a....
[conn=162, chan=130] Received channel close
[conn=162, pktid=1183] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=162, chan=130, pktid=1184] Sent MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 00                                   a....
[conn=164, chan=130, pktid=1313] Received MSG_CHANNEL_WINDOW_ADJUST (93), 9 bytes
  00000000: 5d 00 00 00 82 00 20 00 00                       ]..... ..
[conn=164, chan=130] Received window adjust of 2097152 bytes, new window 2097152
[conn=164, chan=130, pktid=1314] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 82                                   c....
[conn=164, chan=130, pktid=1315] Received MSG_CHANNEL_DATA (94), 26 bytes
  00000000: 5e 00 00 00 82 00 00 00 11 2f 75 73 72 2f 62 69  ^......../usr/bi
  00000010: 6e 2f 70 79 74 68 6f 6e 33 0a                    n/python3.
[conn=164, chan=130] Received 17 data bytes
[conn=164, chan=130, pktid=1316] Received MSG_CHANNEL_EOF (96), 5 bytes
  00000000: 60 00 00 00 82                                   `....
[conn=164, chan=130] Received EOF
[conn=164, chan=130, pktid=1317] Received MSG_CHANNEL_REQUEST (98), 25 bytes
  00000000: 62 00 00 00 82 00 00 00 0b 65 78 69 74 2d 73 74  b........exit-st
  00000010: 61 74 75 73 00 00 00 00 00                       atus.....
[conn=164, chan=130] Received exit status 0
[conn=164, chan=130, pktid=1318] Received MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 82                                   a....
[conn=164, chan=130] Received channel close
[conn=164, pktid=1183] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=164, chan=130, pktid=1184] Sent MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 00                                   a....
[conn=80, chan=133] Set write buffer limits: low-water=16384, high-water=65536
[conn=80, chan=133] Requesting new SSH session
[conn=80, chan=133]   Initial recv window 2097152, packet size 32768
[conn=80, pktid=1197] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=80, chan=133, pktid=1198] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 85  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=88, chan=133] Set write buffer limits: low-water=16384, high-water=65536
[conn=88, chan=133] Requesting new SSH session
[conn=88, chan=133]   Initial recv window 2097152, packet size 32768
[conn=88, pktid=1197] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=88, chan=133, pktid=1198] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 85  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=84, chan=133] Set write buffer limits: low-water=16384, high-water=65536
[conn=84, chan=133] Requesting new SSH session
[conn=84, chan=133]   Initial recv window 2097152, packet size 32768
[conn=84, pktid=1199] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=84, chan=133, pktid=1200] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 85  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=162, chan=130] Reading from channel started
[conn=162, chan=130] Channel closed
[conn=164, chan=130] Reading from channel started
[conn=80, pktid=1335] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 85 00 00 00 00 00 00 00 00 00 00 80  [...............
  00000010: 00                                               .
[conn=80, chan=133]   Initial send window 0, packet size 32768
[conn=88, pktid=1336] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 85 00 00 00 00 00 00 00 00 00 00 80  [...............
  00000010: 00                                               .
[conn=88, chan=133]   Initial send window 0, packet size 32768
[conn=84, pktid=1332] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 85 00 00 00 00 00 00 00 00 00 00 80  [...............
  00000010: 00                                               .
[conn=84, chan=133]   Initial send window 0, packet size 32768
[conn=164, chan=130] Channel closed
[conn=80, chan=133]   Command: sudo /usr/bin/python3 /var/lib/ceph/f148c330-47c9-11ec-9f19-1dfe2cdc6a6d/cephadm.4fd955d0dbf0cdd56104823ef3e950293acaef76608980c02d37021d4c51ee67 check-host
[conn=80, pktid=1199] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=80, chan=133, pktid=1200] Sent MSG_CHANNEL_REQUEST (98), 174 bytes
  00000000: 62 00 00 00 00 00 00 00 04 65 78 65 63 01 00 00  b........exec...
  00000010: 00 9c 73 75 64 6f 20 2f 75 73 72 2f 62 69 6e 2f  ..sudo /usr/bin/
  00000020: 70 79 74 68 6f 6e 33 20 2f 76 61 72 2f 6c 69 62  python3 /var/lib
  00000030: 2f 63 65 70 68 2f 66 31 34 38 63 33 33 30 2d 34  /ceph/f148c330-4
  00000040: 37 63 39 2d 31 31 65 63 2d 39 66 31 39 2d 31 64  7c9-11ec-9f19-1d
  00000050: 66 65 32 63 64 63 36 61 36 64 2f 63 65 70 68 61  fe2cdc6a6d/cepha
  00000060: 64 6d 2e 34 66 64 39 35 35 64 30 64 62 66 30 63  dm.4fd955d0dbf0c
  00000070: 64 64 35 36 31 30 34 38 32 33 65 66 33 65 39 35  dd56104823ef3e95
  00000080: 30 32 39 33 61 63 61 65 66 37 36 36 30 38 39 38  0293acaef7660898
  00000090: 30 63 30 32 64 33 37 30 32 31 64 34 63 35 31 65  0c02d37021d4c51e
  000000a0: 65 36 37 20 63 68 65 63 6b 2d 68 6f 73 74        e67 check-host
[conn=88, chan=133]   Command: sudo /usr/bin/python3 /var/lib/ceph/f148c330-47c9-11ec-9f19-1dfe2cdc6a6d/cephadm.4fd955d0dbf0cdd56104823ef3e950293acaef76608980c02d37021d4c51ee67 check-host
[conn=88, pktid=1199] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=88, chan=133, pktid=1200] Sent MSG_CHANNEL_REQUEST (98), 174 bytes
  00000000: 62 00 00 00 00 00 00 00 04 65 78 65 63 01 00 00  b........exec...
  00000010: 00 9c 73 75 64 6f 20 2f 75 73 72 2f 62 69 6e 2f  ..sudo /usr/bin/
  00000020: 70 79 74 68 6f 6e 33 20 2f 76 61 72 2f 6c 69 62  python3 /var/lib
  00000030: 2f 63 65 70 68 2f 66 31 34 38 63 33 33 30 2d 34  /ceph/f148c330-4
  00000040: 37 63 39 2d 31 31 65 63 2d 39 66 31 39 2d 31 64  7c9-11ec-9f19-1d
  00000050: 66 65 32 63 64 63 36 61 36 64 2f 63 65 70 68 61  fe2cdc6a6d/cepha
  00000060: 64 6d 2e 34 66 64 39 35 35 64 30 64 62 66 30 63  dm.4fd955d0dbf0c
  00000070: 64 64 35 36 31 30 34 38 32 33 65 66 33 65 39 35  dd56104823ef3e95
  00000080: 30 32 39 33 61 63 61 65 66 37 36 36 30 38 39 38  0293acaef7660898
  00000090: 30 63 30 32 64 33 37 30 32 31 64 34 63 35 31 65  0c02d37021d4c51e
  000000a0: 65 36 37 20 63 68 65 63 6b 2d 68 6f 73 74        e67 check-host
[conn=84, chan=133]   Command: sudo /usr/bin/python3 /var/lib/ceph/f148c330-47c9-11ec-9f19-1dfe2cdc6a6d/cephadm.4fd955d0dbf0cdd56104823ef3e950293acaef76608980c02d37021d4c51ee67 check-host
[conn=84, pktid=1201] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=84, chan=133, pktid=1202] Sent MSG_CHANNEL_REQUEST (98), 174 bytes
  00000000: 62 00 00 00 00 00 00 00 04 65 78 65 63 01 00 00  b........exec...
  00000010: 00 9c 73 75 64 6f 20 2f 75 73 72 2f 62 69 6e 2f  ..sudo /usr/bin/
  00000020: 70 79 74 68 6f 6e 33 20 2f 76 61 72 2f 6c 69 62  python3 /var/lib
  00000030: 2f 63 65 70 68 2f 66 31 34 38 63 33 33 30 2d 34  /ceph/f148c330-4
  00000040: 37 63 39 2d 31 31 65 63 2d 39 66 31 39 2d 31 64  7c9-11ec-9f19-1d
  00000050: 66 65 32 63 64 63 36 61 36 64 2f 63 65 70 68 61  fe2cdc6a6d/cepha
  00000060: 64 6d 2e 34 66 64 39 35 35 64 30 64 62 66 30 63  dm.4fd955d0dbf0c
  00000070: 64 64 35 36 31 30 34 38 32 33 65 66 33 65 39 35  dd56104823ef3e95
  00000080: 30 32 39 33 61 63 61 65 66 37 36 36 30 38 39 38  0293acaef7660898
  00000090: 30 63 30 32 64 33 37 30 32 31 64 34 63 35 31 65  0c02d37021d4c51e
  000000a0: 65 36 37 20 63 68 65 63 6b 2d 68 6f 73 74        e67 check-host
[conn=80, chan=133, pktid=1336] Received MSG_CHANNEL_WINDOW_ADJUST (93), 9 bytes
  00000000: 5d 00 00 00 85 00 20 00 00                       ]..... ..
[conn=80, chan=133] Received window adjust of 2097152 bytes, new window 2097152
[conn=80, chan=133, pktid=1337] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 85                                   c....
[conn=88, chan=133, pktid=1337] Received MSG_CHANNEL_WINDOW_ADJUST (93), 9 bytes
  00000000: 5d 00 00 00 85 00 20 00 00                       ]..... ..
[conn=88, chan=133] Received window adjust of 2097152 bytes, new window 2097152
[conn=88, chan=133, pktid=1338] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 85                                   c....
[conn=84, chan=133, pktid=1333] Received MSG_CHANNEL_WINDOW_ADJUST (93), 9 bytes
  00000000: 5d 00 00 00 85 00 20 00 00                       ]..... ..
[conn=84, chan=133] Received window adjust of 2097152 bytes, new window 2097152
[conn=84, chan=133, pktid=1334] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 85                                   c....
[conn=162, chan=131] Set write buffer limits: low-water=16384, high-water=65536
[conn=162, chan=131] Requesting new SSH session
[conn=162, chan=131]   Initial recv window 2097152, packet size 32768
[conn=162, pktid=1185] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=162, chan=131, pktid=1186] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 83  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=80, chan=133] Reading from channel started
[conn=88, chan=133] Reading from channel started
[conn=164, chan=131] Set write buffer limits: low-water=16384, high-water=65536
[conn=164, chan=131] Requesting new SSH session
[conn=164, chan=131]   Initial recv window 2097152, packet size 32768
[conn=164, pktid=1185] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=164, chan=131, pktid=1186] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 83  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=84, chan=133] Reading from channel started
[conn=162, pktid=1317] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 83 00 00 00 00 00 00 00 00 00 00 80  [...............
  00000010: 00                                               .
[conn=162, chan=131]   Initial send window 0, packet size 32768
[conn=162, chan=131]   Command: sudo /usr/bin/python3 /var/lib/ceph/f148c330-47c9-11ec-9f19-1dfe2cdc6a6d/cephadm.4fd955d0dbf0cdd56104823ef3e950293acaef76608980c02d37021d4c51ee67 check-host
[conn=162, pktid=1187] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=162, chan=131, pktid=1188] Sent MSG_CHANNEL_REQUEST (98), 174 bytes
  00000000: 62 00 00 00 00 00 00 00 04 65 78 65 63 01 00 00  b........exec...
  00000010: 00 9c 73 75 64 6f 20 2f 75 73 72 2f 62 69 6e 2f  ..sudo /usr/bin/
  00000020: 70 79 74 68 6f 6e 33 20 2f 76 61 72 2f 6c 69 62  python3 /var/lib
  00000030: 2f 63 65 70 68 2f 66 31 34 38 63 33 33 30 2d 34  /ceph/f148c330-4
  00000040: 37 63 39 2d 31 31 65 63 2d 39 66 31 39 2d 31 64  7c9-11ec-9f19-1d
  00000050: 66 65 32 63 64 63 36 61 36 64 2f 63 65 70 68 61  fe2cdc6a6d/cepha
  00000060: 64 6d 2e 34 66 64 39 35 35 64 30 64 62 66 30 63  dm.4fd955d0dbf0c
  00000070: 64 64 35 36 31 30 34 38 32 33 65 66 33 65 39 35  dd56104823ef3e95
  00000080: 30 32 39 33 61 63 61 65 66 37 36 36 30 38 39 38  0293acaef7660898
  00000090: 30 63 30 32 64 33 37 30 32 31 64 34 63 35 31 65  0c02d37021d4c51e
  000000a0: 65 36 37 20 63 68 65 63 6b 2d 68 6f 73 74        e67 check-host
[conn=164, pktid=1319] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 83 00 00 00 00 00 00 00 00 00 00 80  [...............
  00000010: 00                                               .
[conn=164, chan=131]   Initial send window 0, packet size 32768
[conn=164, chan=131]   Command: sudo /usr/bin/python3 /var/lib/ceph/f148c330-47c9-11ec-9f19-1dfe2cdc6a6d/cephadm.4fd955d0dbf0cdd56104823ef3e950293acaef76608980c02d37021d4c51ee67 check-host
[conn=164, pktid=1187] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=164, chan=131, pktid=1188] Sent MSG_CHANNEL_REQUEST (98), 174 bytes
  00000000: 62 00 00 00 00 00 00 00 04 65 78 65 63 01 00 00  b........exec...
  00000010: 00 9c 73 75 64 6f 20 2f 75 73 72 2f 62 69 6e 2f  ..sudo /usr/bin/
  00000020: 70 79 74 68 6f 6e 33 20 2f 76 61 72 2f 6c 69 62  python3 /var/lib
  00000030: 2f 63 65 70 68 2f 66 31 34 38 63 33 33 30 2d 34  /ceph/f148c330-4
  00000040: 37 63 39 2d 31 31 65 63 2d 39 66 31 39 2d 31 64  7c9-11ec-9f19-1d
  00000050: 66 65 32 63 64 63 36 61 36 64 2f 63 65 70 68 61  fe2cdc6a6d/cepha
  00000060: 64 6d 2e 34 66 64 39 35 35 64 30 64 62 66 30 63  dm.4fd955d0dbf0c
  00000070: 64 64 35 36 31 30 34 38 32 33 65 66 33 65 39 35  dd56104823ef3e95
  00000080: 30 32 39 33 61 63 61 65 66 37 36 36 30 38 39 38  0293acaef7660898
  00000090: 30 63 30 32 64 33 37 30 32 31 64 34 63 35 31 65  0c02d37021d4c51e
  000000a0: 65 36 37 20 63 68 65 63 6b 2d 68 6f 73 74        e67 check-host
[conn=162, chan=131, pktid=1318] Received MSG_CHANNEL_WINDOW_ADJUST (93), 9 bytes
  00000000: 5d 00 00 00 83 00 20 00 00                       ]..... ..
[conn=162, chan=131] Received window adjust of 2097152 bytes, new window 2097152
[conn=162, chan=131, pktid=1319] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 83                                   c....
[conn=164, chan=131, pktid=1320] Received MSG_CHANNEL_WINDOW_ADJUST (93), 9 bytes
  00000000: 5d 00 00 00 83 00 20 00 00                       ]..... ..
[conn=164, chan=131] Received window adjust of 2097152 bytes, new window 2097152
[conn=164, chan=131, pktid=1321] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 83                                   c....
[conn=162, chan=131] Reading from channel started
[conn=164, chan=131] Reading from channel started
[conn=80, chan=133, pktid=1338] Received MSG_CHANNEL_EXTENDED_DATA (95), 49 bytes
  00000000: 5f 00 00 00 85 00 00 00 01 00 00 00 24 64 6f 63  _...........$doc
  00000010: 6b 65 72 20 28 2f 75 73 72 2f 62 69 6e 2f 64 6f  ker (/usr/bin/do
  00000020: 63 6b 65 72 29 20 69 73 20 70 72 65 73 65 6e 74  cker) is present
  00000030: 0a                                               .
[conn=80, chan=133] Received 36 data bytes from STDERR
[conn=80, chan=133, pktid=1339] Received MSG_CHANNEL_EXTENDED_DATA (95), 54 bytes
  00000000: 5f 00 00 00 85 00 00 00 01 00 00 00 29 73 79 73  _...........)sys
  00000010: 74 65 6d 63 74 6c 20 69 73 20 70 72 65 73 65 6e  temctl is presen
  00000020: 74 0a 6c 76 63 72 65 61 74 65 20 69 73 20 70 72  t.lvcreate is pr
  00000030: 65 73 65 6e 74 0a                                esent.
[conn=80, chan=133] Received 41 data bytes from STDERR
[conn=80, chan=133, pktid=1340] Received MSG_CHANNEL_EXTENDED_DATA (95), 56 bytes
  00000000: 5f 00 00 00 85 00 00 00 01 00 00 00 2b 55 6e 69  _...........+Uni
  00000010: 74 20 63 68 72 6f 6e 79 2e 73 65 72 76 69 63 65  t chrony.service
  00000020: 20 69 73 20 65 6e 61 62 6c 65 64 20 61 6e 64 20   is enabled and 
  00000030: 72 75 6e 6e 69 6e 67 0a                          running.
[conn=80, chan=133] Received 43 data bytes from STDERR
[conn=80, chan=133, pktid=1341] Received MSG_CHANNEL_EXTENDED_DATA (95), 27 bytes
  00000000: 5f 00 00 00 85 00 00 00 01 00 00 00 0e 48 6f 73  _............Hos
  00000010: 74 20 6c 6f 6f 6b 73 20 4f 4b 0a                 t looks OK.
[conn=80, chan=133] Received 14 data bytes from STDERR
[conn=80, chan=133, pktid=1342] Received MSG_CHANNEL_REQUEST (98), 25 bytes
  00000000: 62 00 00 00 85 00 00 00 0b 65 78 69 74 2d 73 74  b........exit-st
  00000010: 61 74 75 73 00 00 00 00 00                       atus.....
[conn=80, chan=133] Received exit status 0
[conn=80, chan=133, pktid=1343] Received MSG_CHANNEL_EOF (96), 5 bytes
  00000000: 60 00 00 00 85                                   `....
[conn=80, chan=133] Received EOF
[conn=80, chan=133, pktid=1344] Received MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 85                                   a....
[conn=80, chan=133] Received channel close
[conn=80, pktid=1201] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=80, chan=133, pktid=1202] Sent MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 00                                   a....
[conn=88, chan=133, pktid=1339] Received MSG_CHANNEL_EXTENDED_DATA (95), 70 bytes
  00000000: 5f 00 00 00 85 00 00 00 01 00 00 00 39 64 6f 63  _...........9doc
  00000010: 6b 65 72 20 28 2f 75 73 72 2f 62 69 6e 2f 64 6f  ker (/usr/bin/do
  00000020: 63 6b 65 72 29 20 69 73 20 70 72 65 73 65 6e 74  cker) is present
  00000030: 0a 73 79 73 74 65 6d 63 74 6c 20 69 73 20 70 72  .systemctl is pr
  00000040: 65 73 65 6e 74 0a                                esent.
[conn=88, chan=133] Received 57 data bytes from STDERR
[conn=88, chan=133, pktid=1340] Received MSG_CHANNEL_EXTENDED_DATA (95), 33 bytes
  00000000: 5f 00 00 00 85 00 00 00 01 00 00 00 14 6c 76 63  _............lvc
  00000010: 72 65 61 74 65 20 69 73 20 70 72 65 73 65 6e 74  reate is present
  00000020: 0a                                               .
[conn=88, chan=133] Received 20 data bytes from STDERR
[conn=88, chan=133, pktid=1341] Received MSG_CHANNEL_EXTENDED_DATA (95), 56 bytes
  00000000: 5f 00 00 00 85 00 00 00 01 00 00 00 2b 55 6e 69  _...........+Uni
  00000010: 74 20 63 68 72 6f 6e 79 2e 73 65 72 76 69 63 65  t chrony.service
  00000020: 20 69 73 20 65 6e 61 62 6c 65 64 20 61 6e 64 20   is enabled and 
  00000030: 72 75 6e 6e 69 6e 67 0a                          running.
[conn=88, chan=133] Received 43 data bytes from STDERR
[conn=88, chan=133, pktid=1342] Received MSG_CHANNEL_EXTENDED_DATA (95), 27 bytes
  00000000: 5f 00 00 00 85 00 00 00 01 00 00 00 0e 48 6f 73  _............Hos
  00000010: 74 20 6c 6f 6f 6b 73 20 4f 4b 0a                 t looks OK.
[conn=88, chan=133] Received 14 data bytes from STDERR
[conn=88, chan=133, pktid=1343] Received MSG_CHANNEL_EOF (96), 5 bytes
  00000000: 60 00 00 00 85                                   `....
[conn=88, chan=133] Received EOF
[conn=88, chan=133, pktid=1344] Received MSG_CHANNEL_REQUEST (98), 25 bytes
  00000000: 62 00 00 00 85 00 00 00 0b 65 78 69 74 2d 73 74  b........exit-st
  00000010: 61 74 75 73 00 00 00 00 00                       atus.....
[conn=88, chan=133] Received exit status 0
[conn=88, chan=133, pktid=1345] Received MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 85                                   a....
[conn=88, chan=133] Received channel close
[conn=88, pktid=1201] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=88, chan=133, pktid=1202] Sent MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 00                                   a....
[conn=84, chan=133, pktid=1335] Received MSG_CHANNEL_EXTENDED_DATA (95), 70 bytes
  00000000: 5f 00 00 00 85 00 00 00 01 00 00 00 39 64 6f 63  _...........9doc
  00000010: 6b 65 72 20 28 2f 75 73 72 2f 62 69 6e 2f 64 6f  ker (/usr/bin/do
  00000020: 63 6b 65 72 29 20 69 73 20 70 72 65 73 65 6e 74  cker) is present
  00000030: 0a 73 79 73 74 65 6d 63 74 6c 20 69 73 20 70 72  .systemctl is pr
  00000040: 65 73 65 6e 74 0a                                esent.
[conn=84, chan=133] Received 57 data bytes from STDERR
[conn=84, chan=133, pktid=1336] Received MSG_CHANNEL_EXTENDED_DATA (95), 33 bytes
  00000000: 5f 00 00 00 85 00 00 00 01 00 00 00 14 6c 76 63  _............lvc
  00000010: 72 65 61 74 65 20 69 73 20 70 72 65 73 65 6e 74  reate is present
  00000020: 0a                                               .
[conn=84, chan=133] Received 20 data bytes from STDERR
[conn=84, chan=133, pktid=1337] Received MSG_CHANNEL_EXTENDED_DATA (95), 56 bytes
  00000000: 5f 00 00 00 85 00 00 00 01 00 00 00 2b 55 6e 69  _...........+Uni
  00000010: 74 20 63 68 72 6f 6e 79 2e 73 65 72 76 69 63 65  t chrony.service
  00000020: 20 69 73 20 65 6e 61 62 6c 65 64 20 61 6e 64 20   is enabled and 
  00000030: 72 75 6e 6e 69 6e 67 0a                          running.
[conn=84, chan=133] Received 43 data bytes from STDERR
[conn=84, chan=133, pktid=1338] Received MSG_CHANNEL_EXTENDED_DATA (95), 27 bytes
  00000000: 5f 00 00 00 85 00 00 00 01 00 00 00 0e 48 6f 73  _............Hos
  00000010: 74 20 6c 6f 6f 6b 73 20 4f 4b 0a                 t looks OK.
[conn=84, chan=133] Received 14 data bytes from STDERR
[conn=84, chan=133, pktid=1339] Received MSG_CHANNEL_REQUEST (98), 25 bytes
  00000000: 62 00 00 00 85 00 00 00 0b 65 78 69 74 2d 73 74  b........exit-st
  00000010: 61 74 75 73 00 00 00 00 00                       atus.....
[conn=84, chan=133] Received exit status 0
[conn=84, chan=133, pktid=1340] Received MSG_CHANNEL_EOF (96), 5 bytes
  00000000: 60 00 00 00 85                                   `....
[conn=84, chan=133] Received EOF
[conn=84, chan=133, pktid=1341] Received MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 85                                   a....
[conn=84, chan=133] Received channel close
[conn=84, pktid=1203] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=84, chan=133, pktid=1204] Sent MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 00                                   a....
[conn=162, chan=131, pktid=1320] Received MSG_CHANNEL_EXTENDED_DATA (95), 70 bytes
  00000000: 5f 00 00 00 83 00 00 00 01 00 00 00 39 64 6f 63  _...........9doc
  00000010: 6b 65 72 20 28 2f 75 73 72 2f 62 69 6e 2f 64 6f  ker (/usr/bin/do
  00000020: 63 6b 65 72 29 20 69 73 20 70 72 65 73 65 6e 74  cker) is present
  00000030: 0a 73 79 73 74 65 6d 63 74 6c 20 69 73 20 70 72  .systemctl is pr
  00000040: 65 73 65 6e 74 0a                                esent.
[conn=162, chan=131] Received 57 data bytes from STDERR
[conn=162, chan=131, pktid=1321] Received MSG_CHANNEL_EXTENDED_DATA (95), 33 bytes
  00000000: 5f 00 00 00 83 00 00 00 01 00 00 00 14 6c 76 63  _............lvc
  00000010: 72 65 61 74 65 20 69 73 20 70 72 65 73 65 6e 74  reate is present
  00000020: 0a                                               .
[conn=162, chan=131] Received 20 data bytes from STDERR
[conn=162, chan=131, pktid=1322] Received MSG_CHANNEL_EXTENDED_DATA (95), 56 bytes
  00000000: 5f 00 00 00 83 00 00 00 01 00 00 00 2b 55 6e 69  _...........+Uni
  00000010: 74 20 63 68 72 6f 6e 79 2e 73 65 72 76 69 63 65  t chrony.service
  00000020: 20 69 73 20 65 6e 61 62 6c 65 64 20 61 6e 64 20   is enabled and 
  00000030: 72 75 6e 6e 69 6e 67 0a                          running.
[conn=162, chan=131] Received 43 data bytes from STDERR
[conn=162, chan=131, pktid=1323] Received MSG_CHANNEL_EXTENDED_DATA (95), 27 bytes
  00000000: 5f 00 00 00 83 00 00 00 01 00 00 00 0e 48 6f 73  _............Hos
  00000010: 74 20 6c 6f 6f 6b 73 20 4f 4b 0a                 t looks OK.
[conn=162, chan=131] Received 14 data bytes from STDERR
[conn=162, chan=131, pktid=1324] Received MSG_CHANNEL_REQUEST (98), 25 bytes
  00000000: 62 00 00 00 83 00 00 00 0b 65 78 69 74 2d 73 74  b........exit-st
  00000010: 61 74 75 73 00 00 00 00 00                       atus.....
[conn=162, chan=131] Received exit status 0
[conn=162, chan=131, pktid=1325] Received MSG_CHANNEL_EOF (96), 5 bytes
  00000000: 60 00 00 00 83                                   `....
[conn=162, chan=131] Received EOF
[conn=162, chan=131, pktid=1326] Received MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 83                                   a....
[conn=162, chan=131] Received channel close
[conn=162, pktid=1189] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=162, chan=131, pktid=1190] Sent MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 00                                   a....
[conn=164, chan=131, pktid=1322] Received MSG_CHANNEL_EXTENDED_DATA (95), 49 bytes
  00000000: 5f 00 00 00 83 00 00 00 01 00 00 00 24 64 6f 63  _...........$doc
  00000010: 6b 65 72 20 28 2f 75 73 72 2f 62 69 6e 2f 64 6f  ker (/usr/bin/do
  00000020: 63 6b 65 72 29 20 69 73 20 70 72 65 73 65 6e 74  cker) is present
  00000030: 0a                                               .
[conn=164, chan=131] Received 36 data bytes from STDERR
[conn=164, chan=131, pktid=1323] Received MSG_CHANNEL_EXTENDED_DATA (95), 54 bytes
  00000000: 5f 00 00 00 83 00 00 00 01 00 00 00 29 73 79 73  _...........)sys
  00000010: 74 65 6d 63 74 6c 20 69 73 20 70 72 65 73 65 6e  temctl is presen
  00000020: 74 0a 6c 76 63 72 65 61 74 65 20 69 73 20 70 72  t.lvcreate is pr
  00000030: 65 73 65 6e 74 0a                                esent.
[conn=164, chan=131] Received 41 data bytes from STDERR
[conn=164, chan=131, pktid=1324] Received MSG_CHANNEL_EXTENDED_DATA (95), 56 bytes
  00000000: 5f 00 00 00 83 00 00 00 01 00 00 00 2b 55 6e 69  _...........+Uni
  00000010: 74 20 63 68 72 6f 6e 79 2e 73 65 72 76 69 63 65  t chrony.service
  00000020: 20 69 73 20 65 6e 61 62 6c 65 64 20 61 6e 64 20   is enabled and 
  00000030: 72 75 6e 6e 69 6e 67 0a                          running.
[conn=164, chan=131] Received 43 data bytes from STDERR
[conn=164, chan=131, pktid=1325] Received MSG_CHANNEL_EXTENDED_DATA (95), 27 bytes
  00000000: 5f 00 00 00 83 00 00 00 01 00 00 00 0e 48 6f 73  _............Hos
  00000010: 74 20 6c 6f 6f 6b 73 20 4f 4b 0a                 t looks OK.
[conn=164, chan=131] Received 14 data bytes from STDERR
[conn=164, chan=131, pktid=1326] Received MSG_CHANNEL_REQUEST (98), 25 bytes
  00000000: 62 00 00 00 83 00 00 00 0b 65 78 69 74 2d 73 74  b........exit-st
  00000010: 61 74 75 73 00 00 00 00 00                       atus.....
[conn=164, chan=131] Received exit status 0
[conn=164, chan=131, pktid=1327] Received MSG_CHANNEL_EOF (96), 5 bytes
  00000000: 60 00 00 00 83                                   `....
[conn=164, chan=131] Received EOF
[conn=164, chan=131, pktid=1328] Received MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 83                                   a....
[conn=164, chan=131] Received channel close
[conn=164, pktid=1189] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=164, chan=131, pktid=1190] Sent MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 00                                   a....
[conn=80, chan=133] Channel closed
[conn=88, chan=133] Channel closed
[conn=84, chan=133] Channel closed
[conn=162, chan=131] Channel closed
[conn=164, chan=131] Channel closed
[conn=158, chan=130] Set write buffer limits: low-water=16384, high-water=65536
[conn=158, chan=130] Requesting new SSH session
[conn=158, chan=130]   Initial recv window 2097152, packet size 32768
[conn=158, pktid=1179] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=158, chan=130, pktid=1180] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 82  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=158, pktid=1313] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 82 00 00 00 00 00 00 00 00 00 00 80  [...............
  00000010: 00                                               .
[conn=158, chan=130]   Initial send window 0, packet size 32768
[conn=158, chan=130]   Command: sudo which python3
[conn=158, pktid=1181] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=158, chan=130, pktid=1182] Sent MSG_CHANNEL_REQUEST (98), 36 bytes
  00000000: 62 00 00 00 00 00 00 00 04 65 78 65 63 01 00 00  b........exec...
  00000010: 00 12 73 75 64 6f 20 77 68 69 63 68 20 70 79 74  ..sudo which pyt
  00000020: 68 6f 6e 33                                      hon3
[conn=158, chan=130, pktid=1314] Received MSG_CHANNEL_WINDOW_ADJUST (93), 9 bytes
  00000000: 5d 00 00 00 82 00 20 00 00                       ]..... ..
[conn=158, chan=130] Received window adjust of 2097152 bytes, new window 2097152
[conn=158, chan=130, pktid=1315] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 82                                   c....
[conn=158, chan=130, pktid=1316] Received MSG_CHANNEL_DATA (94), 26 bytes
  00000000: 5e 00 00 00 82 00 00 00 11 2f 75 73 72 2f 62 69  ^......../usr/bi
  00000010: 6e 2f 70 79 74 68 6f 6e 33 0a                    n/python3.
[conn=158, chan=130] Received 17 data bytes
[conn=158, chan=130, pktid=1317] Received MSG_CHANNEL_EOF (96), 5 bytes
  00000000: 60 00 00 00 82                                   `....
[conn=158, chan=130] Received EOF
[conn=158, chan=130, pktid=1318] Received MSG_CHANNEL_REQUEST (98), 25 bytes
  00000000: 62 00 00 00 82 00 00 00 0b 65 78 69 74 2d 73 74  b........exit-st
  00000010: 61 74 75 73 00 00 00 00 00                       atus.....
[conn=158, chan=130] Received exit status 0
[conn=158, chan=130, pktid=1319] Received MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 82                                   a....
[conn=158, chan=130] Received channel close
[conn=158, pktid=1183] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=158, chan=130, pktid=1184] Sent MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 00                                   a....
[conn=158, chan=130] Reading from channel started
[conn=158, chan=130] Channel closed
[conn=158, chan=131] Set write buffer limits: low-water=16384, high-water=65536
[conn=158, chan=131] Requesting new SSH session
[conn=158, chan=131]   Initial recv window 2097152, packet size 32768
[conn=158, pktid=1185] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=158, chan=131, pktid=1186] Sent MSG_CHANNEL_OPEN (90), 24 bytes
  00000000: 5a 00 00 00 07 73 65 73 73 69 6f 6e 00 00 00 83  Z....session....
  00000010: 00 20 00 00 00 00 80 00                          . ......
[conn=158, pktid=1320] Received MSG_CHANNEL_OPEN_CONFIRMATION (91), 17 bytes
  00000000: 5b 00 00 00 83 00 00 00 00 00 00 00 00 00 00 80  [...............
  00000010: 00                                               .
[conn=158, chan=131]   Initial send window 0, packet size 32768
[conn=158, chan=131]   Command: sudo /usr/bin/python3 /var/lib/ceph/f148c330-47c9-11ec-9f19-1dfe2cdc6a6d/cephadm.4fd955d0dbf0cdd56104823ef3e950293acaef76608980c02d37021d4c51ee67 check-host
[conn=158, pktid=1187] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=158, chan=131, pktid=1188] Sent MSG_CHANNEL_REQUEST (98), 174 bytes
  00000000: 62 00 00 00 00 00 00 00 04 65 78 65 63 01 00 00  b........exec...
  00000010: 00 9c 73 75 64 6f 20 2f 75 73 72 2f 62 69 6e 2f  ..sudo /usr/bin/
  00000020: 70 79 74 68 6f 6e 33 20 2f 76 61 72 2f 6c 69 62  python3 /var/lib
  00000030: 2f 63 65 70 68 2f 66 31 34 38 63 33 33 30 2d 34  /ceph/f148c330-4
  00000040: 37 63 39 2d 31 31 65 63 2d 39 66 31 39 2d 31 64  7c9-11ec-9f19-1d
  00000050: 66 65 32 63 64 63 36 61 36 64 2f 63 65 70 68 61  fe2cdc6a6d/cepha
  00000060: 64 6d 2e 34 66 64 39 35 35 64 30 64 62 66 30 63  dm.4fd955d0dbf0c
  00000070: 64 64 35 36 31 30 34 38 32 33 65 66 33 65 39 35  dd56104823ef3e95
  00000080: 30 32 39 33 61 63 61 65 66 37 36 36 30 38 39 38  0293acaef7660898
  00000090: 30 63 30 32 64 33 37 30 32 31 64 34 63 35 31 65  0c02d37021d4c51e
  000000a0: 65 36 37 20 63 68 65 63 6b 2d 68 6f 73 74        e67 check-host
[conn=158, chan=131, pktid=1321] Received MSG_CHANNEL_WINDOW_ADJUST (93), 9 bytes
  00000000: 5d 00 00 00 83 00 20 00 00                       ]..... ..
[conn=158, chan=131] Received window adjust of 2097152 bytes, new window 2097152
[conn=158, chan=131, pktid=1322] Received MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 83                                   c....
[conn=158, chan=131] Reading from channel started
[conn=158, chan=131, pktid=1323] Received MSG_CHANNEL_EXTENDED_DATA (95), 70 bytes
  00000000: 5f 00 00 00 83 00 00 00 01 00 00 00 39 64 6f 63  _...........9doc
  00000010: 6b 65 72 20 28 2f 75 73 72 2f 62 69 6e 2f 64 6f  ker (/usr/bin/do
  00000020: 63 6b 65 72 29 20 69 73 20 70 72 65 73 65 6e 74  cker) is present
  00000030: 0a 73 79 73 74 65 6d 63 74 6c 20 69 73 20 70 72  .systemctl is pr
  00000040: 65 73 65 6e 74 0a                                esent.
[conn=158, chan=131] Received 57 data bytes from STDERR
[conn=158, chan=131, pktid=1324] Received MSG_CHANNEL_EXTENDED_DATA (95), 33 bytes
  00000000: 5f 00 00 00 83 00 00 00 01 00 00 00 14 6c 76 63  _............lvc
  00000010: 72 65 61 74 65 20 69 73 20 70 72 65 73 65 6e 74  reate is present
  00000020: 0a                                               .
[conn=158, chan=131] Received 20 data bytes from STDERR
[conn=158, chan=131, pktid=1325] Received MSG_CHANNEL_EXTENDED_DATA (95), 56 bytes
  00000000: 5f 00 00 00 83 00 00 00 01 00 00 00 2b 55 6e 69  _...........+Uni
  00000010: 74 20 63 68 72 6f 6e 79 2e 73 65 72 76 69 63 65  t chrony.service
  00000020: 20 69 73 20 65 6e 61 62 6c 65 64 20 61 6e 64 20   is enabled and 
  00000030: 72 75 6e 6e 69 6e 67 0a                          running.
[conn=158, chan=131] Received 43 data bytes from STDERR
[conn=158, chan=131, pktid=1326] Received MSG_CHANNEL_EXTENDED_DATA (95), 27 bytes
  00000000: 5f 00 00 00 83 00 00 00 01 00 00 00 0e 48 6f 73  _............Hos
  00000010: 74 20 6c 6f 6f 6b 73 20 4f 4b 0a                 t looks OK.
[conn=158, chan=131] Received 14 data bytes from STDERR
[conn=158, chan=131, pktid=1327] Received MSG_CHANNEL_EOF (96), 5 bytes
  00000000: 60 00 00 00 83                                   `....
[conn=158, chan=131] Received EOF
[conn=158, chan=131, pktid=1328] Received MSG_CHANNEL_REQUEST (98), 25 bytes
  00000000: 62 00 00 00 83 00 00 00 0b 65 78 69 74 2d 73 74  b........exit-st
  00000010: 61 74 75 73 00 00 00 00 00                       atus.....
[conn=158, chan=131] Received exit status 0
[conn=158, chan=131, pktid=1329] Received MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 83                                   a....
[conn=158, chan=131] Received channel close
[conn=158, pktid=1189] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
[conn=158, chan=131, pktid=1190] Sent MSG_CHANNEL_CLOSE (97), 5 bytes
  00000000: 61 00 00 00 00                                   a....
[conn=158, chan=131] Channel closed
[conn=1688] Connection failure: [Errno 104] Connection reset by peer
[conn=1688] Aborting connection

The error message is long enough it makes it difficult to use the ceph health detail command at all.

Similar messages will also appear within the mgr logs which can also make them harder to read and can make the logs quite large over time.

It seems to be the message that includes "possibly because python3 is not installed there" that is the worst offender.

History

#1 Updated by Melissa Li about 1 year ago

  • Assignee set to Melissa Li

#2 Updated by Sebastian Wagner about 1 year ago

  • Status changed from New to Resolved

Also available in: Atom PDF