Ceph » Orchestrator

This may not be reproducible since I haven't seen it anywhere else. Will update this Tracker if I see any more instances.
Later in the logs, the KeyError becomes a variation of the one logged here, i.e. "enp3s0f1".

Description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}

Failure Reason: hit max job timeout

/a/lflores-2022-01-05_19:04:35-rados-wip-lflores-mgr-rocksdb-distro-default-smithi/6596713

2022-01-05T22:03:19.766 INFO:journalctl@ceph.mon.smithi074.smithi074.stdout:Jan 05 22:03:19 smithi074 conmon[35475]: cephadm 2022-01-05T22:03:18.752250+0000 mgr.smithi022.icnyet (mgr.14164) 1172 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Traceback (most recent call last):
2022-01-05T22:03:19.766 INFO:journalctl@ceph.mon.smithi074.smithi074.stdout:Jan 05 22:03:19 smithi074 conmon[35475]:   File "/var/lib/ceph/82c92cb2-6e6e-11ec-8c32-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in <module>
2022-01-05T22:03:19.766 INFO:journalctl@ceph.mon.smithi074.smithi074.stdout:Jan 05 22:03:19 smithi074 conmon[35475]:     main()
2022-01-05T22:03:19.767 INFO:journalctl@ceph.mon.smithi074.smithi074.stdout:Jan 05 22:03:19 smithi074 conmon[35475]:   File "/var/lib/ceph/82c92cb2-6e6e-11ec-8c32-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main
2022-01-05T22:03:19.767 INFO:journalctl@ceph.mon.smithi074.smithi074.stdout:Jan 05 22:03:19 smithi074 conmon[35475]:     r = ctx.func(ctx)
2022-01-05T22:03:19.767 INFO:journalctl@ceph.mon.smithi074.smithi074.stdout:Jan 05 22:03:19 smithi074 conmon[35475]:   File "/var/lib/ceph/82c92cb2-6e6e-11ec-8c32-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4651, in command_list_networks
2022-01-05T22:03:19.767 INFO:journalctl@ceph.mon.smithi074.smithi074.stdout:Jan 05 22:03:19 smithi074 conmon[35475]:     r = list_networks(ctx)
2022-01-05T22:03:20.042 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]: cluster 2022-01-05T22:03:18.019387+0000
2022-01-05T22:03:20.042 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]: mgr.smithi022.icnyet (mgr.14164) 1171 : cluster [DBG] pgmap v1009: 65 pgs: 65 active+clean; 309 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s
2022-01-05T22:03:20.042 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]: cephadm 2022-01-05T22:03:18.752250+0000 mgr.smithi022.icnyet (mgr.14164) 1172 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Traceback (most recent call last):
2022-01-05T22:03:20.043 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]:   File "/var/lib/ceph/82c92cb2-6e6e-11ec-8c32-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in <module>
2022-01-05T22:03:20.043 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]:     main()
2022-01-05T22:03:20.043 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]:   File "/var/lib/ceph/82c92cb2-6e6e-11ec-8c32-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main
2022-01-05T22:03:20.044 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]:     r = ctx.func(ctx)
2022-01-05T22:03:20.044 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]:   File "/var/lib/ceph/82c92cb2-6e6e-11ec-8c32-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4651, in command_list_networks
2022-01-05T22:03:20.044 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]:     r = list_networks(ctx)
2022-01-05T22:03:20.044 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]:   File "/var/lib/ceph/82c92cb2-6e6e-11ec-8c32-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4572, in list_networks
2022-01-05T22:03:20.045 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]:     res.update(_list_ipv6_networks(ctx))
2022-01-05T22:03:20.045 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]:   File "/var/lib/ceph/82c92cb2-6e6e-11ec-8c32-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4608, in _list_ipv6_networks
2022-01-05T22:03:20.045 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]:     return _parse_ipv6_route(routes, ips)
2022-01-05T22:03:20.045 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]:   File "/var/lib/ceph/82c92cb2-6e6e-11ec-8c32-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4644, in _parse_ipv6_route
2022-01-05T22:03:20.046 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]:     r[net[0]][iface].append(ip)
2022-01-05T22:03:20.046 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Jan 05 22:03:19 smithi022 conmon[31810]: KeyError: 'ens1f0'