Ceph » Orchestrator

/a/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629126

2024-03-29T04:52:23.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe206bda-ed85-11ee-95d0-87774f69a715 -e sha1=8da23624bc17fe0b1f797227e72a0db0ce7821ba -- bash -c 'ceph orch ps'
...
2024-03-29T04:57:26.132 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 29 04:57:25 smithi067 ceph-fe206bda-ed85-11ee-95d0-87774f69a715-osd-0[47753]: 2024-03-29T04:57:25.746+0000 7fbad1e3f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.194:6804 osd.3 since back 2024-03-29T04:47:03.191908+0000 front 2024-03-29T04:47:03.191953+0000 (oldest deadline 2024-03-29T04:47:26.691849+0000)
2024-03-29T04:57:26.133 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 29 04:57:25 smithi067 ceph-fe206bda-ed85-11ee-95d0-87774f69a715-osd-0[47753]: 2024-03-29T04:57:25.746+0000 7fbad1e3f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.194:6812 osd.4 since back 2024-03-29T04:47:03.191923+0000 front 2024-03-29T04:47:03.191940+0000 (oldest deadline 2024-03-29T04:47:26.691849+0000)
2024-03-29T04:57:26.133 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 29 04:57:25 smithi067 ceph-fe206bda-ed85-11ee-95d0-87774f69a715-osd-0[47753]: 2024-03-29T04:57:25.746+0000 7fbad1e3f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.194:6820 osd.5 since back 2024-03-29T04:47:03.191972+0000 front 2024-03-29T04:47:03.191962+0000 (oldest deadline 2024-03-29T04:47:26.691849+0000)
2024-03-29T04:57:26.133 INFO:journalctl@ceph.osd.0.smithi067.stdout:Mar 29 04:57:25 smithi067 ceph-fe206bda-ed85-11ee-95d0-87774f69a715-osd-0[47753]: 2024-03-29T04:57:25.747+0000 7fbad1e3f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24293.0:6364 3.12 3:4facbde1:::10000004725.00000000:head [write 0~2532 in=2532b] snapc 1=[] ondisk+write+known_if_redirected e46)
2024-03-29T04:57:26.133 INFO:journalctl@ceph.osd.2.smithi067.stdout:Mar 29 04:57:25 smithi067 ceph-fe206bda-ed85-11ee-95d0-87774f69a715-osd-2[63213]: 2024-03-29T04:57:25.834+0000 7f13c3942700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.194:6804 osd.3 since back 2024-03-29T04:47:05.530417+0000 front 2024-03-29T04:47:05.530437+0000 (oldest deadline 2024-03-29T04:47:31.430334+0000)
2024-03-29T04:57:26.133 INFO:journalctl@ceph.osd.2.smithi067.stdout:Mar 29 04:57:25 smithi067 ceph-fe206bda-ed85-11ee-95d0-87774f69a715-osd-2[63213]: 2024-03-29T04:57:25.834+0000 7f13c3942700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.194:6812 osd.4 since back 2024-03-29T04:47:05.530362+0000 front 2024-03-29T04:47:05.530382+0000 (oldest deadline 2024-03-29T04:47:31.430334+0000)
2024-03-29T04:57:26.133 INFO:journalctl@ceph.osd.2.smithi067.stdout:Mar 29 04:57:25 smithi067 ceph-fe206bda-ed85-11ee-95d0-87774f69a715-osd-2[63213]: 2024-03-29T04:57:25.834+0000 7f13c3942700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.194:6820 osd.5 since back 2024-03-29T04:47:05.530452+0000 front 2024-03-29T04:47:05.530407+0000 (oldest deadline 2024-03-29T04:47:31.430334+0000)
2024-03-29T04:57:26.133 INFO:journalctl@ceph.osd.2.smithi067.stdout:Mar 29 04:57:25 smithi067 ceph-fe206bda-ed85-11ee-95d0-87774f69a715-osd-2[63213]: 2024-03-29T04:57:25.835+0000 7f13c3942700 -1 osd.2 46 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.24293.0:6359 3.8 3:1140aa8a:::10000004721.00000000:head [write 0~1391 in=1391b] snapc 1=[] ondisk+write+known_if_redirected e46)
2024-03-29T04:57:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-03-29T04:57:26.253 ERROR:teuthology.run_tasks:Saw exception from tasks.