Project

General

Profile

Actions
Copy link

Bug #62225

open

pacific upgrade test fails on 'ceph versions | jq -e' command

Added by Laura Flores 9 months ago. Updated 3 months ago.

Status:
New
Priority:
Normal
Assignee:
Adam King
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2023-07-19_14:33:14-rados-wip-yuri11-testing-2023-07-18-0927-pacific-distro-default-smithi/7343484

2023-07-19T16:00:36.785 INFO:teuthology.orchestra.run.smithi131.stdout:{
2023-07-19T16:00:36.786 INFO:teuthology.orchestra.run.smithi131.stdout:    "mon": {
2023-07-19T16:00:36.786 INFO:teuthology.orchestra.run.smithi131.stdout:        "ceph version 16.2.13-355-ge2018073 (e20180733e4b085d1abe55675eb8c828b0bfedec) pacific (stable)": 3
2023-07-19T16:00:36.786 INFO:teuthology.orchestra.run.smithi131.stdout:    },
2023-07-19T16:00:36.786 INFO:teuthology.orchestra.run.smithi131.stdout:    "mgr": {
2023-07-19T16:00:36.786 INFO:teuthology.orchestra.run.smithi131.stdout:        "ceph version 16.2.13-355-ge2018073 (e20180733e4b085d1abe55675eb8c828b0bfedec) pacific (stable)": 2
2023-07-19T16:00:36.787 INFO:teuthology.orchestra.run.smithi131.stdout:    },
2023-07-19T16:00:36.787 INFO:teuthology.orchestra.run.smithi131.stdout:    "osd": {
2023-07-19T16:00:36.787 INFO:teuthology.orchestra.run.smithi131.stdout:        "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 6,
2023-07-19T16:00:36.787 INFO:teuthology.orchestra.run.smithi131.stdout:        "ceph version 16.2.13-355-ge2018073 (e20180733e4b085d1abe55675eb8c828b0bfedec) pacific (stable)": 2
2023-07-19T16:00:36.787 INFO:teuthology.orchestra.run.smithi131.stdout:    },
2023-07-19T16:00:36.787 INFO:teuthology.orchestra.run.smithi131.stdout:    "mds": {},
2023-07-19T16:00:36.788 INFO:teuthology.orchestra.run.smithi131.stdout:    "rgw": {
2023-07-19T16:00:36.788 INFO:teuthology.orchestra.run.smithi131.stdout:        "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 2
2023-07-19T16:00:36.788 INFO:teuthology.orchestra.run.smithi131.stdout:    },
2023-07-19T16:00:36.788 INFO:teuthology.orchestra.run.smithi131.stdout:    "overall": {
2023-07-19T16:00:36.789 INFO:teuthology.orchestra.run.smithi131.stdout:        "ceph version 15.2.0 (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 8,
2023-07-19T16:00:36.789 INFO:teuthology.orchestra.run.smithi131.stdout:        "ceph version 16.2.13-355-ge2018073 (e20180733e4b085d1abe55675eb8c828b0bfedec) pacific (stable)": 7
2023-07-19T16:00:36.789 INFO:teuthology.orchestra.run.smithi131.stdout:    }
2023-07-19T16:00:36.789 INFO:teuthology.orchestra.run.smithi131.stdout:}
2023-07-19T16:00:37.154 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9b35ffc-264b-11ee-9b34-001a4aab830c -e sha1=e20180733e4b085d1abe55675eb8c828b0bfedec -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"''
2023-07-19T16:00:37.405 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jul 19 16:00:37 smithi131 ceph-d9b35ffc-264b-11ee-9b34-001a4aab830c-mon-c[134106]: cluster 2023-07-19T16:00:36.083212+0000 mon.a (mon.0) 348 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in
2023-07-19T16:00:37.406 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jul 19 16:00:37 smithi131 ceph-d9b35ffc-264b-11ee-9b34-001a4aab830c-mon-c[134106]: audit 2023-07-19T16:00:36.789908+0000 mon.c (mon.1) 5 : audit [DBG] from='client.34158 172.21.15.131:0/1205111914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch
2023-07-19T16:00:37.406 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jul 19 16:00:37 smithi131 ceph-d9b35ffc-264b-11ee-9b34-001a4aab830c-mon-c[134106]: cluster 2023-07-19T16:00:37.035832+0000 mon.a (mon.0) 349 : cluster [WRN] Health check update: Degraded data redundancy: 82/606 objects degraded (13.531%), 21 pgs degraded (PG_DEGRADED)
2023-07-19T16:00:37.406 INFO:journalctl@ceph.mon.a.smithi131.stdout:Jul 19 16:00:37 smithi131 ceph-d9b35ffc-264b-11ee-9b34-001a4aab830c-mon-a[132122]: cluster 2023-07-19T16:00:36.083212+0000 mon.a (mon.0) 348 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in
2023-07-19T16:00:37.406 INFO:journalctl@ceph.mon.a.smithi131.stdout:Jul 19 16:00:37 smithi131 ceph-d9b35ffc-264b-11ee-9b34-001a4aab830c-mon-a[132122]: audit 2023-07-19T16:00:36.789908+0000 mon.c (mon.1) 5 : audit [DBG] from='client.34158 172.21.15.131:0/1205111914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch
2023-07-19T16:00:37.406 INFO:journalctl@ceph.mon.a.smithi131.stdout:Jul 19 16:00:37 smithi131 ceph-d9b35ffc-264b-11ee-9b34-001a4aab830c-mon-a[132122]: cluster 2023-07-19T16:00:37.035832+0000 mon.a (mon.0) 349 : cluster [WRN] Health check update: Degraded data redundancy: 82/606 objects degraded (13.531%), 21 pgs degraded (PG_DEGRADED)
2023-07-19T16:00:37.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jul 19 16:00:37 smithi190 ceph-d9b35ffc-264b-11ee-9b34-001a4aab830c-mon-b[125600]: cluster 2023-07-19T16:00:36.083212+0000 mon.a (mon.0) 348 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in
2023-07-19T16:00:37.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jul 19 16:00:37 smithi190 ceph-d9b35ffc-264b-11ee-9b34-001a4aab830c-mon-b[125600]: audit 2023-07-19T16:00:36.789908+0000 mon.c (mon.1) 5 : audit [DBG] from='client.34158 172.21.15.131:0/1205111914' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch
2023-07-19T16:00:37.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jul 19 16:00:37 smithi190 ceph-d9b35ffc-264b-11ee-9b34-001a4aab830c-mon-b[125600]: cluster 2023-07-19T16:00:37.035832+0000 mon.a (mon.0) 349 : cluster [WRN] Health check update: Degraded data redundancy: 82/606 objects degraded (13.531%), 21 pgs degraded (PG_DEGRADED)
2023-07-19T16:00:38.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jul 19 16:00:38 smithi190 ceph-d9b35ffc-264b-11ee-9b34-001a4aab830c-mon-b[125600]: cluster 2023-07-19T16:00:36.475656+0000 mgr.y (mgr.24871) 91 : cluster [DBG] pgmap v48: 129 pgs: 4 peering, 20 active+undersized+degraded, 32 active+undersized, 73 active+clean; 7.0 KiB data, 17 MiB used, 709 GiB / 715 GiB avail; 76/606 objects degraded (12.541%)
2023-07-19T16:00:38.446 INFO:journalctl@ceph.mon.a.smithi131.stdout:Jul 19 16:00:38 smithi131 ceph-d9b35ffc-264b-11ee-9b34-001a4aab830c-mon-a[132122]: cluster 2023-07-19T16:00:36.475656+0000 mgr.y (mgr.24871) 91 : cluster [DBG] pgmap v48: 129 pgs: 4 peering, 20 active+undersized+degraded, 32 active+undersized, 73 active+clean; 7.0 KiB data, 17 MiB used, 709 GiB / 715 GiB avail; 76/606 objects degraded (12.541%)
2023-07-19T16:00:38.447 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jul 19 16:00:38 smithi131 ceph-d9b35ffc-264b-11ee-9b34-001a4aab830c-mon-c[134106]: cluster 2023-07-19T16:00:36.475656+0000 mgr.y (mgr.24871) 91 : cluster [DBG] pgmap v48: 129 pgs: 4 peering, 20 active+undersized+degraded, 32 active+undersized, 73 active+clean; 7.0 KiB data, 17 MiB used, 709 GiB / 715 GiB avail; 76/606 objects degraded (12.541%)
2023-07-19T16:00:38.531 INFO:teuthology.orchestra.run.smithi131.stdout:false
2023-07-19T16:00:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1
2023-07-19T16:00:38.880 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):

Related issues 1 (0 open — 1 closed)

Has duplicate RADOS - Bug #64105: pacific upgrade test fails on 'ceph versions | jq -e' commandDuplicate

Actions
Actions
Copy link
 #1

Updated by Laura Flores 9 months ago

/a/yuriw-2023-07-20_14:33:07-rados-wip-yuri11-testing-2023-07-18-0927-pacific-distro-default-smithi/7344744

Actions
Copy link
 #2

Updated by Radoslaw Zarzynski 9 months ago

Was it seen on octopus2quincy or pacific2quincy? Asking b/c pacific goes EOL.

Actions
Copy link
 #3

Updated by Laura Flores 9 months ago

Radoslaw Zarzynski wrote:

Was it seen on octopus2quincy or pacific2quincy? Asking b/c pacific goes EOL.

Octopus to pacific

Actions
Copy link
 #4

Updated by Laura Flores 6 months ago

/a/lflores-2023-11-01_18:38:59-rados-wip-yuri5-testing-2023-10-24-0737-pacific-distro-default-smithi/7443320

Actions
Copy link
 #5

Updated by Laura Flores 5 months ago

/a/lflores-2023-11-21_19:51:47-rados-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7463960

Actions
Copy link
 #6

Updated by Nitzan Mordechai 4 months ago

/a/yuriw-2023-12-26_16:04:49-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7501361
/a/yuriw-2023-12-26_16:04:49-rados-wip-yuri5-testing-2023-12-15-0747-pacific-distro-default-smithi/7501390

Actions
Copy link
 #7

Updated by Nitzan Mordechai 4 months ago

/a/yuriw-2023-12-27_21:01:10-rados-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/7502422
/a/yuriw-2023-12-27_21:01:10-rados-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/7502371
/a/yuriw-2023-12-27_21:01:10-rados-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/7502338
/a/yuriw-2023-12-27_21:01:10-rados-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/7502351

Actions
Copy link
 #8

Updated by Nitzan Mordechai 4 months ago

Its not because of degraded pgs, After the upgrade of the mgr, the orch module didn't start or we didn't wait enough so it will start.

if we are adding: ceph mgr module enable orchestrator --force
that problem will be solved, but then we will hit an issue with migration_current that has a value of 5 instead of 3.
/a/nmordech-2023-12-28_14:42:08-rados:cephadm:mgr-nfs-upgrade-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/7502863

Actions
Copy link
 #9

Updated by Laura Flores 4 months ago

Looks like a similar issue:
/a/yuriw-2024-01-08_16:16:59-rados-wip-yuri-testing-2024-01-03-0851-pacific-distro-default-smithi/7509105

Actions
Copy link
 #10

Updated by Prashant D 4 months ago

  • Copied to Bug #64105: pacific upgrade test fails on 'ceph versions | jq -e' command added
Actions
Copy link
 #11

Updated by Prashant D 4 months ago

  • Copied to deleted (Bug #64105: pacific upgrade test fails on 'ceph versions | jq -e' command)
Actions
Copy link
 #12

Updated by Laura Flores 4 months ago

  • Has duplicate Bug #64105: pacific upgrade test fails on 'ceph versions | jq -e' command added
Actions
Copy link
 #13

Updated by Laura Flores 4 months ago

  • Subject changed from pacific upgrade test fails when upgrading OSDs due to degraded pgs to pacific upgrade test fails on 'ceph versions | jq -e' command
Actions
Copy link
 #14

Updated by Laura Flores 4 months ago

/a/yuriw-2024-01-19_23:17:02-rados-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7524136

Actions
Copy link
 #15

Updated by Laura Flores 3 months ago

/a/yuriw-2024-01-23_19:22:22-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7529535

Actions
Copy link
 #16

Updated by Laura Flores 3 months ago

/a/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531973
/a/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7532188
/a/yuriw-2024-01-25_17:23:47-rados-pacific-release-distro-default-smithi/7531799

Actions
Copy link
 #17

Updated by Laura Flores 3 months ago

/a/yuriw-2024-02-01_21:25:50-rados-wip-yuri2-testing-2024-02-01-0939-pacific-distro-default-smithi/7542413

Actions
Copy link
 #18

Updated by Radoslaw Zarzynski 3 months ago

  • Assignee set to Adam King

A cephadm test issue?

Actions
Copy link

Also available in: Atom PDF