Bug #57449
Updated by Patrick Donnelly over 1 year ago
<pre>
2022-08-31T17:11:51.057 INFO:journalctl@ceph.mon.c.smithi174.stdout:Aug 31 17:11:50 smithi174 ceph-mon[103875]: executing refresh((['smithi073', 'smithi153', 'smithi174'],)) failed.
2022-08-31T17:11:51.057 INFO:journalctl@ceph.mon.c.smithi174.stdout: Traceback (most recent call last):
2022-08-31T17:11:51.058 INFO:journalctl@ceph.mon.c.smithi174.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 143, in _execute_command
2022-08-31T17:11:51.058 INFO:journalctl@ceph.mon.c.smithi174.stdout: r = await conn.run('sudo true', check=True, timeout=5)
2022-08-31T17:11:51.058 INFO:journalctl@ceph.mon.c.smithi174.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 3637, in run
2022-08-31T17:11:51.058 INFO:journalctl@ceph.mon.c.smithi174.stdout: return await process.wait(check, timeout)
2022-08-31T17:11:51.059 INFO:journalctl@ceph.mon.c.smithi174.stdout: File "/lib/python3.6/site-packages/asyncssh/process.py", line 1252, in wait
2022-08-31T17:11:51.059 INFO:journalctl@ceph.mon.c.smithi174.stdout: stderr_data) from None
2022-08-31T17:11:51.059 INFO:journalctl@ceph.mon.c.smithi174.stdout: asyncssh.process.TimeoutError
</pre>
/ceph/teuthology-archive/pdonnell-2022-08-31_03:04:35-fs:workload-wip-pdonnell-testing-20220831.010330-distro-default-smithi/7002272/teuthology.log
and
<pre>
2022-08-31T17:12:49.762+0000 7f75dc618700 0 [cephadm DEBUG cephadm.inventory] Host "smithi073" marked as offline. Skipping daemon refresh
2022-08-31T17:12:49.762+0000 7f75dc618700 0 [cephadm DEBUG cephadm.inventory] Host "smithi073" marked as offline. Skipping gather facts refresh
2022-08-31T17:12:49.762+0000 7f75dc618700 0 [cephadm DEBUG cephadm.inventory] Host "smithi073" marked as offline. Skipping network refresh
2022-08-31T17:12:49.762+0000 7f75dc618700 0 [cephadm DEBUG cephadm.inventory] Host "smithi073" marked as offline. Skipping device refresh
2022-08-31T17:12:49.762+0000 7f75dc618700 0 [cephadm DEBUG cephadm.inventory] Host "smithi073" marked as offline. Skipping osdspec preview refresh
2022-08-31T17:12:49.762+0000 7f75dc618700 0 [cephadm DEBUG cephadm.inventory] Host "smithi073" marked as offline. Skipping autotune
2022-08-31T17:12:49.762+0000 7f75dc618700 0 [cephadm INFO cephadm.serve] Removing smithi073:/var/lib/ceph/4313329e-294d-11ed-8431-001a4aab830c/config/ceph.conf
2022-08-31T17:12:49.762+0000 7f75dc618700 0 log_channel(cephadm) log [INF] : Removing smithi073:/var/lib/ceph/4313329e-294d-11ed-8431-001a4aab830c/config/ceph.conf
</pre>
/ceph/teuthology-archive/pdonnell-2022-08-31_03:04:35-fs:workload-wip-pdonnell-testing-20220831.010330-distro-default-smithi/7002272/remote/smithi073/log/4313329e-294d-11ed-8431-001a4aab830c/ceph-mgr.x.log.gz
One interesting thing here is that the ceph-mgr is removing the host it's on! The remove smithi073 is the localhost.
See also a few more in
https://pulpito.ceph.com/pdonnell-2022-08-31_03:04:35-fs:workload-wip-pdonnell-testing-20220831.010330-distro-default-smithi/
-
All with the "ffsb" workunit.- See also
/ceph/teuthology-archive/pdonnell-2022-09-02_01:35:33-fs:workload-wip-pdonnell-testing-20220901.192929-distro-default-smithi/7006546/teuthology.log workunit.