Project

General

Profile

Bug #58659

mds_upgrade_sequence: failure when deploying node-exporter

Added by Laura Flores about 1 year ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/lflores-2023-02-07_03:56:23-rados-wip-yuri4-testing-2023-02-07-0246-pacific-distro-default-smithi/7152652

description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools
conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down
pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes
3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}

2023-02-07T04:34:08.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: cluster 2023-02-07T04:34:06.775504+0000 mgr.smithi099.rczixy (mgr.14162) 129 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail
2023-02-07T04:34:08.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: cephadm 2023-02-07T04:34:07.495605+0000 mgr.smithi099.rczixy (mgr.14162) 130 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi150 ...
2023-02-07T04:34:08.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: Verifying port 9100 ...
2023-02-07T04:34:08.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: Non-zero exit code 1 from systemctl start ceph-efce3a36-a69f-11ed-bc20-001a4aab830c@node-exporter.smithi150
2023-02-07T04:34:08.663 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: systemctl: stderr Job for ceph-efce3a36-a69f-11ed-bc20-001a4aab830c@node-exporter.smithi150.service failed because a timeout was exceeded.
2023-02-07T04:34:08.663 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: systemctl: stderr See "systemctl status ceph-efce3a36-a69f-11ed-bc20-001a4aab830c@node-exporter.smithi150.service" and "journalctl -xe" for details.
2023-02-07T04:34:08.663 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: Traceback (most recent call last):
2023-02-07T04:34:08.664 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in <module>
2023-02-07T04:34:08.664 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     main()
2023-02-07T04:34:08.665 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main
2023-02-07T04:34:08.665 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     r = ctx.func(ctx)
2023-02-07T04:34:08.665 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image
2023-02-07T04:34:08.666 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     return func(ctx)
2023-02-07T04:34:08.666 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy
2023-02-07T04:34:08.666 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     ports=daemon_ports)
2023-02-07T04:34:08.667 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon
2023-02-07T04:34:08.667 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     c, osd_fsid=osd_fsid, ports=ports)
2023-02-07T04:34:08.668 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units
2023-02-07T04:34:08.668 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     call_throws(ctx, ['systemctl', 'start', unit_name])
2023-02-07T04:34:08.668 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws
2023-02-07T04:34:08.669 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     raise RuntimeError('Failed command: %s' % ' '.join(command))
2023-02-07T04:34:08.669 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: RuntimeError: Failed command: systemctl start ceph-efce3a36-a69f-11ed-bc20-001a4aab830c@node-exporter.smithi150

History

#1 Updated by Kamoltat (Junior) Sirivadhna 4 months ago

https://pulpito.ceph.com/yuriw-2023-11-14_20:31:57-fs-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458082/

2023-11-15T02:11:28.455+0000 7f74730d5700 10 mon.smithi124@0(leader).log v135  logging 2023-11-15T02:11:27.657024+0000 mgr.smithi124.fxsvlx (mgr.14162) 85 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi124 ...
Verifying port 9100 ...
Non-zero exit code 1 from systemctl start ceph-b7d116f0-835b-11ee-8dbe-212e2dc638e7@node-exporter.smithi124
systemctl: stderr Job for ceph-b7d116f0-835b-11ee-8dbe-212e2dc638e7@node-exporter.smithi124.service failed because a timeout was exceeded.
systemctl: stderr See "systemctl status ceph-b7d116f0-835b-11ee-8dbe-212e2dc638e7@node-exporter.smithi124.service" and "journalctl -xe" for details.
Traceback (most recent call last):
  File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in <module>
    main()
  File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main
    r = ctx.func(ctx)
  File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image
    return func(ctx)
  File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy
    ports=daemon_ports)
  File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon
    c, osd_fsid=osd_fsid, ports=ports)
  File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units
    call_throws(ctx, ['systemctl', 'start', unit_name])
  File "/var/lib/ceph/b7d116f0-835b-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws
    raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: systemctl start ceph-b7d116f0-835b-11ee-8dbe-212e2dc638e7@node-exporter.smithi124
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection
    yield (conn, connr)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm
    code, '\n'.join(err)))
orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi124 ...

#2 Updated by Laura Flores 3 months ago

/a/yuriw-2024-01-08_16:16:59-rados-wip-yuri-testing-2024-01-03-0851-pacific-distro-default-smithi/7509109

#3 Updated by Laura Flores 2 months ago

/a/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524259

#4 Updated by Laura Flores 2 months ago

/a/yuriw-2024-01-23_19:22:22-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7529712

#5 Updated by Laura Flores about 2 months ago

/a/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532277
/a/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532508

Also available in: Atom PDF