Bug #58659
mds_upgrade_sequence: failure when deploying node-exporter
0%
Description
/a/lflores-2023-02-07_03:56:23-rados-wip-yuri4-testing-2023-02-07-0246-pacific-distro-default-smithi/7152652
description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools
conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down
pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes
3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
2023-02-07T04:34:08.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: cluster 2023-02-07T04:34:06.775504+0000 mgr.smithi099.rczixy (mgr.14162) 129 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail
2023-02-07T04:34:08.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: cephadm 2023-02-07T04:34:07.495605+0000 mgr.smithi099.rczixy (mgr.14162) 130 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi150 ...
2023-02-07T04:34:08.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: Verifying port 9100 ...
2023-02-07T04:34:08.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: Non-zero exit code 1 from systemctl start ceph-efce3a36-a69f-11ed-bc20-001a4aab830c@node-exporter.smithi150
2023-02-07T04:34:08.663 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: systemctl: stderr Job for ceph-efce3a36-a69f-11ed-bc20-001a4aab830c@node-exporter.smithi150.service failed because a timeout was exceeded.
2023-02-07T04:34:08.663 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: systemctl: stderr See "systemctl status ceph-efce3a36-a69f-11ed-bc20-001a4aab830c@node-exporter.smithi150.service" and "journalctl -xe" for details.
2023-02-07T04:34:08.663 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: Traceback (most recent call last):
2023-02-07T04:34:08.664 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in <module>
2023-02-07T04:34:08.664 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: main()
2023-02-07T04:34:08.665 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main
2023-02-07T04:34:08.665 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: r = ctx.func(ctx)
2023-02-07T04:34:08.665 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image
2023-02-07T04:34:08.666 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: return func(ctx)
2023-02-07T04:34:08.666 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy
2023-02-07T04:34:08.666 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: ports=daemon_ports)
2023-02-07T04:34:08.667 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon
2023-02-07T04:34:08.667 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: c, osd_fsid=osd_fsid, ports=ports)
2023-02-07T04:34:08.668 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units
2023-02-07T04:34:08.668 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: call_throws(ctx, ['systemctl', 'start', unit_name])
2023-02-07T04:34:08.668 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws
2023-02-07T04:34:08.669 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: raise RuntimeError('Failed command: %s' % ' '.join(command))
2023-02-07T04:34:08.669 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: RuntimeError: Failed command: systemctl start ceph-efce3a36-a69f-11ed-bc20-001a4aab830c@node-exporter.smithi150
History
#1 Updated by Kamoltat (Junior) Sirivadhna 4 months ago
2023-11-15T02:11:28.455+0000 7f74730d5700 10 mon.smithi124@0(leader).log v135 logging 2023-11-15T02:11:27.657024+0000 mgr.smithi124.fxsvlx (mgr.14162) 85 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi124 ... Verifying port 9100 ... Non-zero exit code 1 from systemctl start ceph-b7d116f0-835b-11ee-8dbe-212e2dc638e7@node-exporter.smithi124 systemctl: stderr Job for ceph-b7d116f0-835b-11ee-8dbe-212e2dc638e7@node-exporter.smithi124.service failed because a timeout was exceeded. systemctl: stderr See "systemctl status ceph-b7d116f0-835b-11ee-8dbe-212e2dc638e7@node-exporter.smithi124.service" and "journalctl -xe" for details. Traceback (most recent call last): File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in <module> main() File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main r = ctx.func(ctx) File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image return func(ctx) File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy ports=daemon_ports) File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon c, osd_fsid=osd_fsid, ports=ports) File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units call_throws(ctx, ['systemctl', 'start', unit_name]) File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: systemctl start ceph-b7d116f0-835b-11ee-8dbe-212e2dc638e7@node-exporter.smithi124 Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection yield (conn, connr) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm code, '\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi124 ...
#2 Updated by Laura Flores 3 months ago
/a/yuriw-2024-01-08_16:16:59-rados-wip-yuri-testing-2024-01-03-0851-pacific-distro-default-smithi/7509109
#3 Updated by Laura Flores 2 months ago
/a/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259
#4 Updated by Laura Flores 2 months ago
/a/yuriw-2024-01-23_19:22:22-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7529712
#5 Updated by Laura Flores about 2 months ago
/a/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532277
/a/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532508