Infrastructure

/a/yuriw-2023-03-14_20:10:47-rados-wip-yuri-testing-2023-03-14-0714-reef-distro-default-smithi/7207025

72.21.15.5:6806 osd.0 ever on either front or back, first ping sent 2023-03-16T20:41:22.885372+0000 (oldest deadline 2023-03-16T20:41:42.885372+0000)
2023-03-17T03:35:11.630 INFO:journalctl@ceph.osd.11.smithi171.stdout:Mar 16 20:50:36 smithi171 ceph-82b414ae-c407-11ed-9afb-001a4aab830c-osd-11[128465]: 2023-03-16T20:50:36.751+0000 7fe5f3c23700 -1 osd.11 244 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 ever on either front or back, first ping sent 2023-03-16T20:41:44.988769+0000 (oldest deadline 2023-03-16T20:42:04.988769+0000)
2023-03-17T03:35:11.630 INFO:journalctl@ceph.osd.11.smithi171.stdout:Mar 16 20:50:36 smithi171 ceph-82b414ae-c407-11ed-9afb-001a4aab830c-osd-11[128465]: 2023-03-16T20:50:36.751+0000 7fe5f3c23700 -1 osd.11 244 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 ever on either front or back, first ping sent 2023-03-16T20:41:44.988769+0000 (oldest deadline 2023-03-16T20:42:04.988769+0000)
2023-03-17T03:35:11.630 INFO:journalctl@ceph.osd.11.smithi171.stdout:Mar 16 20:50:36 smithi171 ceph-82b414ae-c407-11ed-9afb-001a4aab830c-osd-11[128465]: 2023-03-16T20:50:36.751+0000 7fe5f3c23700 -1 osd.11 244 heartbeat_check: no reply from 172.21.15.5:6830 osd.3 ever on either front or back, first ping sent 2023-03-16T20:42:44.596501+0000 (oldest deadline 2023-03-16T20:43:04.596501+0000)
2023-03-17T03:35:11.630 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2023-03-17T03:35:11.631 DEBUG:teuthology.orchestra.run:got remote process result: None
2023-03-17T03:35:12.715 INFO:journalctl@ceph.osd.9.smithi171.stdout:Mar 16 21:48:54 smithi171 ceph-82b414ae-c407-11ed-9afb-001a4aab830c-osd-9[119037]: 2023-03-16T21:48:54.224+0000 7fd32a188700 -1 osd.9 244 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 ever on either front or back, first ping sent 2023-03-16T21:46:30.693848+0000 (oldest deadline 2023-03-16T21:46:50.693
2023-03-17T03:35:12.715 INFO:tasks.cephadm.osd.9:Stopped osd.9
2023-03-17T03:35:12.715 INFO:tasks.cephadm.osd.10:Stopping osd.10...
2023-03-17T03:35:12.716 ERROR:tasks.cephadm:Failed to stop "ceph.osd.10" 
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 436, in run
    (host, port) = transport.getpeername()[0:2]
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1841, in getpeername
    return gp()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 560, in getpeername
    return self._sock.getpeername()
OSError: [Errno 107] Transport endpoint is not connected

Seems to be another bug related to https://tracker.ceph.com/issues/59127. Note the repeated "heartbeat_check: no reply" lines, which is a symptom of the https://tracker.ceph.com/issues/59127.