Actions
Bug #62959
opencephadm: staggered upgrade test fails when using agent
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
At some point during staggered upgrade we run
Command failed on smithi017 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 59755a4a-599d-11ee-8db4-212e2dc638e7 -e sha1=e658726abb668aaae1c42e72d314001acd82ce18 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''
which is to check that 7 total daemons are currently upgraded. Right before this `ceph orch ps` is reporting
2023-09-22T23:41:04.766 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-09-22T23:41:04.766 INFO:teuthology.orchestra.run.smithi017.stdout:agent.smithi017 smithi017 running - 28m - - <unknown> <unknown> <unknown> 2023-09-22T23:41:04.766 INFO:teuthology.orchestra.run.smithi017.stdout:agent.smithi164 smithi164 running - 28m - - <unknown> <unknown> <unknown> 2023-09-22T23:41:04.766 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) - 24m 17.7M - 0.25.0 c8568f914cd2 bcd7cace23a9 2023-09-22T23:41:04.766 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi164 *:3000 running (3m) - 24m 88.5M - 9.4.7 2c41d148cca3 290b21fdb49a 2023-09-22T23:41:04.766 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.itnheu smithi017 running (4m) - 23m 61.2M - 3.5 e1d6a67b021e e73fab77de20 2023-09-22T23:41:04.766 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi164 *:8443,9283,8765 running (4m) - 27m 448M - 18.0.0-6358-ge658726a 1d431065767a f7454978cfce 2023-09-22T23:41:04.766 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (14m) - 29m 541M - 18.0.0-6358-ge658726a 1d431065767a aaa4b418534a 2023-09-22T23:41:04.767 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (112s) - 29m 77.9M 2048M 18.0.0-6358-ge658726a 1d431065767a 5158ed731239 2023-09-22T23:41:04.767 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi164 running (2m) - 27m 71.8M 2048M 18.0.0-6358-ge658726a 1d431065767a 338345547b60 2023-09-22T23:41:04.767 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (68s) - 28m 56.4M 2048M 18.0.0-6358-ge658726a 1d431065767a 7287e08b9870 2023-09-22T23:41:04.767 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) - 24m 10.1M - 1.5.0 0da6a335fe13 ad906d0b3773 2023-09-22T23:41:04.767 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi164 *:9100 running (4m) - 24m 19.2M - 1.5.0 0da6a335fe13 7927a4c821b8 2023-09-22T23:41:04.767 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (13s) - 27m 48.2M 4096M 18.0.0-6358-ge658726a 1d431065767a d6500dc023fb 2023-09-22T23:41:04.767 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (27m) - 27m 199M 4096M 17.2.0 e1d6a67b021e 2dd431ff179b 2023-09-22T23:41:04.767 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (26m) - 26m 143M 4096M 17.2.0 e1d6a67b021e 971042890bdf 2023-09-22T23:41:04.767 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (26m) - 26m 224M 4096M 17.2.0 e1d6a67b021e 140201782de3 2023-09-22T23:41:04.768 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi164 running (26m) - 26m 211M 4096M 17.2.0 e1d6a67b021e a9e5272aaf90 2023-09-22T23:41:04.768 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi164 running (26m) - 26m 192M 4096M 17.2.0 e1d6a67b021e 0797cc0911d5 2023-09-22T23:41:04.768 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi164 running (25m) - 25m 161M 4096M 17.2.0 e1d6a67b021e 01e373896323 2023-09-22T23:41:04.768 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi164 running (25m) - 25m 204M 4096M 17.2.0 e1d6a67b021e 3174f2af79ec 2023-09-22T23:41:04.768 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi164 *:9095 running (4m) - 25m 74.2M - 2.43.0 a07b618ecd1d 8fe022d78a23 2023-09-22T23:41:04.768 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.ydekee smithi017 *:8000 running (24m) - 24m 97.5M - 17.2.0 e1d6a67b021e c533c2ae40d3 2023-09-22T23:41:04.768 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi164.cwemma smithi164 *:8000 running (24m) - 24m 97.6M - 17.2.0 e1d6a67b021e be24218971f1
which implies 6 of the 7 daemons we expect to be upgraded have been upgraded.
A bit before this in the logs we can see
2023-09-22T23:40:59.357 INFO:journalctl@ceph.mon.b.smithi164.stdout:Sep 22 23:40:58 smithi164 ceph-mon[180657]: Metadata not up to date on all hosts. Skipping non agent specs 2023-09-22T23:40:59.357 INFO:journalctl@ceph.mon.b.smithi164.stdout:Sep 22 23:40:58 smithi164 ceph-mon[180657]: Metadata not up to date on all hosts. Skipping non agent specs 2023-09-22T23:40:59.357 INFO:journalctl@ceph.mon.b.smithi164.stdout:Sep 22 23:40:58 smithi164 ceph-mon[180657]: Metadata not up to date on all hosts. Skipping non agent specs 2023-09-22T23:40:59.357 INFO:journalctl@ceph.mon.b.smithi164.stdout:Sep 22 23:40:58 smithi164 ceph-mon[180657]: Metadata not up to date on all hosts. Skipping non agent specs 2023-09-22T23:40:59.357 INFO:journalctl@ceph.mon.b.smithi164.stdout:Sep 22 23:40:58 smithi164 ceph-mon[180657]: Upgrade: Finalizing container_image settings 2023-09-22T23:40:59.358 INFO:journalctl@ceph.mon.b.smithi164.stdout:Sep 22 23:40:58 smithi164 ceph-mon[180657]: Upgrade: Complete!
I think upgrades while using the agent, or at least staggered ones, just don't work correctly right now. This was a gap in testing until recently since we used to start most of our upgrade tests from pacific where the agent did not exist.
No data to display
Actions