Project

General

Profile

Actions
Copy link

Bug #58659

open

mds_upgrade_sequence: failure when deploying node-exporter

Added by Laura Flores over 1 year ago. Updated 4 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/lflores-2023-02-07_03:56:23-rados-wip-yuri4-testing-2023-02-07-0246-pacific-distro-default-smithi/7152652

description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools
conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down
pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes
3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}

2023-02-07T04:34:08.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: cluster 2023-02-07T04:34:06.775504+0000 mgr.smithi099.rczixy (mgr.14162) 129 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail
2023-02-07T04:34:08.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: cephadm 2023-02-07T04:34:07.495605+0000 mgr.smithi099.rczixy (mgr.14162) 130 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi150 ...
2023-02-07T04:34:08.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: Verifying port 9100 ...
2023-02-07T04:34:08.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: Non-zero exit code 1 from systemctl start ceph-efce3a36-a69f-11ed-bc20-001a4aab830c@node-exporter.smithi150
2023-02-07T04:34:08.663 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: systemctl: stderr Job for ceph-efce3a36-a69f-11ed-bc20-001a4aab830c@node-exporter.smithi150.service failed because a timeout was exceeded.
2023-02-07T04:34:08.663 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: systemctl: stderr See "systemctl status ceph-efce3a36-a69f-11ed-bc20-001a4aab830c@node-exporter.smithi150.service" and "journalctl -xe" for details.
2023-02-07T04:34:08.663 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: Traceback (most recent call last):
2023-02-07T04:34:08.664 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in <module>
2023-02-07T04:34:08.664 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     main()
2023-02-07T04:34:08.665 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main
2023-02-07T04:34:08.665 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     r = ctx.func(ctx)
2023-02-07T04:34:08.665 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image
2023-02-07T04:34:08.666 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     return func(ctx)
2023-02-07T04:34:08.666 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy
2023-02-07T04:34:08.666 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     ports=daemon_ports)
2023-02-07T04:34:08.667 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon
2023-02-07T04:34:08.667 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     c, osd_fsid=osd_fsid, ports=ports)
2023-02-07T04:34:08.668 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units
2023-02-07T04:34:08.668 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     call_throws(ctx, ['systemctl', 'start', unit_name])
2023-02-07T04:34:08.668 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:   File "/var/lib/ceph/efce3a36-a69f-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws
2023-02-07T04:34:08.669 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]:     raise RuntimeError('Failed command: %s' % ' '.join(command))
2023-02-07T04:34:08.669 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Feb 07 04:34:08 smithi099 ceph-efce3a36-a69f-11ed-bc20-001a4aab830c-mon.smithi099[103544]: RuntimeError: Failed command: systemctl start ceph-efce3a36-a69f-11ed-bc20-001a4aab830c@node-exporter.smithi150
Actions
Copy link

Also available in: Atom PDF