Project

General

Profile

Bug #46254

cephadm upgrade test: exit condition is wrong. we have to wait longer

Added by Sebastian Wagner 5 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
teuthology
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

We're exiting the upgrade loop too early. In this case, the OSD didn't had the time to came up again.

2020-06-26T22:18:16.543+0000 7f64f3fff700 -1 WARNING: all dangerous and experimental features are enabled.
NAME                     HOST       STATUS        REFRESHED  AGE  VERSION               IMAGE NAME                                                          IMAGE ID      CONTAINER ID
alertmanager.smithi107   smithi107  running (6m)  37s ago    8m   0.20.0                prom/alertmanager:v0.20.0                                           0881eb8f169f  59f590bb47e4
crash.smithi025          smithi025  running (7m)  3m ago     7m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  c3e2611b0e61
crash.smithi107          smithi107  running (7m)  37s ago    7m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  7b4672b542f4
grafana.smithi107        smithi107  running (6m)  37s ago    7m   6.6.2                 ceph/ceph-grafana:6.6.2                                             a0dce381714a  3c7ec1c57c9d
mgr.smithi025.uyujka     smithi025  running (7m)  3m ago     7m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  27de5aa9955a
mgr.smithi107.huttci     smithi107  running (9m)  37s ago    9m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  eb8dc48883dd
mon.smithi025            smithi025  running (7m)  3m ago     7m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  89169260c9ba
mon.smithi107            smithi107  running (9m)  37s ago    9m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  489aa98f61ab
node-exporter.smithi025  smithi025  running (7m)  3m ago     7m   0.18.1                prom/node-exporter:v0.18.1                                          e5a616e4b9cf  760b6ed37ada
node-exporter.smithi107  smithi107  running (7m)  37s ago    7m   0.18.1                prom/node-exporter:v0.18.1                                          e5a616e4b9cf  89cbf6d1d7c4
osd.0                    smithi107  running (6m)  37s ago    6m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  58df31321f25
osd.1                    smithi107  running (5m)  37s ago    5m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  8196b4bd7205
osd.2                    smithi107  running (5m)  37s ago    5m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  7744612334e7
osd.3                    smithi107  running (4m)  37s ago    4m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  75953fafb1b4
osd.4                    smithi025  running (4m)  3m ago     4m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  ed9286e28808
osd.5                    smithi025  running (3m)  3m ago     3m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  d31b5b134f81
osd.6                    smithi025  running (3m)  3m ago     3m   16.0.0-2955-g8b33a29  quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  81a8ed38fd46  305707e2bd2b
osd.7                    smithi025  running       3m ago     3m   <unknown>             quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9  <unknown>     <unknown>
prometheus.smithi107     smithi107  running (6m)  37s ago    7m   2.18.1                prom/prometheus:v2.18.1                                             de242295e225  d4d5e7937c37
> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80d969f8-b7f9-11ea-a06d-001a4aab830c -e sha1=8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9 -- bash -c 'ceph versions'
B / 715 GiB avail
2020-06-26T22:18:20.831+0000 7fa965ffb700 -1 WARNING: all dangerous and experimental features are enabled.
{
    "mon": {
        "ceph version 16.0.0-2955-g8b33a29 (8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9) pacific (dev)": 2
    },
    "mgr": {
        "ceph version 16.0.0-2955-g8b33a29 (8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9) pacific (dev)": 2
    },
    "osd": {
        "ceph version 16.0.0-2955-g8b33a29 (8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9) pacific (dev)": 8
    },
    "mds": {},
    "overall": {
        "ceph version 16.0.0-2955-g8b33a29 (8b33a29f3a7f00f8e6f971159aa875abdbdf0dd9) pacific (dev)": 12
    }
}

We need to wait, till osd.7 is back up again.

Also available in: Atom PDF