Bug #46254
cephadm upgrade test: exit condition is wrong. we have to wait longer
Status:
Can't reproduce
Priority:
Low
Assignee:
-
Category:
teuthology
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
We're exiting the upgrade loop too early. In this case, the OSD didn't had the time to came up again.
2020-06-26T22:18:16.543+0000 7f64f3fff700 -1 WARNING: all dangerous and experimental features are enabled. NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID alertmanager.smithi107 smithi107 running (6m) 37s ago 8m 0.20.0 prom/alertmanager:v0.20.0 0881eb8f169f 59f590bb47e4 crash.smithi025 smithi025 running (7m) 3m ago 7m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 c3e2611b0e61 crash.smithi107 smithi107 running (7m) 37s ago 7m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 7b4672b542f4 grafana.smithi107 smithi107 running (6m) 37s ago 7m 6.6.2 ceph/ceph-grafana:6.6.2 a0dce381714a 3c7ec1c57c9d mgr.smithi025.uyujka smithi025 running (7m) 3m ago 7m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 27de5aa9955a mgr.smithi107.huttci smithi107 running (9m) 37s ago 9m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 eb8dc48883dd mon.smithi025 smithi025 running (7m) 3m ago 7m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 89169260c9ba mon.smithi107 smithi107 running (9m) 37s ago 9m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 489aa98f61ab node-exporter.smithi025 smithi025 running (7m) 3m ago 7m 0.18.1 prom/node-exporter:v0.18.1 e5a616e4b9cf 760b6ed37ada node-exporter.smithi107 smithi107 running (7m) 37s ago 7m 0.18.1 prom/node-exporter:v0.18.1 e5a616e4b9cf 89cbf6d1d7c4 osd.0 smithi107 running (6m) 37s ago 6m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 58df31321f25 osd.1 smithi107 running (5m) 37s ago 5m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 8196b4bd7205 osd.2 smithi107 running (5m) 37s ago 5m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 7744612334e7 osd.3 smithi107 running (4m) 37s ago 4m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 75953fafb1b4 osd.4 smithi025 running (4m) 3m ago 4m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 ed9286e28808 osd.5 smithi025 running (3m) 3m ago 3m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 d31b5b134f81 osd.6 smithi025 running (3m) 3m ago 3m 16.0.0-2955-g8b33a29 quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 81a8ed38fd46 305707e2bd2b osd.7 smithi025 running 3m ago 3m <unknown> quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 <unknown> <unknown> prometheus.smithi107 smithi107 running (6m) 37s ago 7m 2.18.1 prom/prometheus:v2.18.1 de242295e225 d4d5e7937c37 > sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d969f8-b7f9-11ea-a06d-001a4aab830c -e sha1=8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 -- bash -c 'ceph versions' B / 715 GiB avail 2020-06-26T22:18:20.831+0000 7fa965ffb700 -1 WARNING: all dangerous and experimental features are enabled. { "mon": { "ceph version 16.0.0-2955-g8b33a29 (8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9) pacific (dev)": 2 }, "mgr": { "ceph version 16.0.0-2955-g8b33a29 (8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9) pacific (dev)": 2 }, "osd": { "ceph version 16.0.0-2955-g8b33a29 (8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9) pacific (dev)": 8 }, "mds": {}, "overall": { "ceph version 16.0.0-2955-g8b33a29 (8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9) pacific (dev)": 12 } }
We need to wait, till osd.7 is back up again.
Related issues
History
#1 Updated by Juan Miguel Olmo MartÃnez about 3 years ago
- Priority changed from Normal to Low
#2 Updated by Sebastian Wagner about 3 years ago
- Related to Bug #46204: cephadm upgrade test: fail if upgrade status is set to error added
#3 Updated by Sebastian Wagner over 2 years ago
- Status changed from New to Can't reproduce