Project

General

Profile

Actions
Copy link

Bug #59604

open

upgrade: unkown ceph version causes upgrade to get stuck

Added by Laura Flores 12 months ago. Updated 12 months ago.

Status:
New
Priority:
Normal
Assignee:
Adam King
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2023-04-26_20:20:35-upgrade:octopus-x-pacific-release-distro-default-smithi/7255267/

2023-04-27T00:27:52.980 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Apr 27 00:27:52 smithi118 bash[149104]: time="2023-04-27T00:27:52Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-2ba77aa2-e491-11ed-9b00-001a4aab830c-mgr.x in 10 seconds, resorting to SIGKILL" 

...

2023-04-27T10:31:26.597 DEBUG:teuthology.exit:Got signal 15; running 1 handler...
2023-04-27T10:31:26.626 DEBUG:teuthology.task.console_log:Killing console logger for smithi114
2023-04-27T10:31:26.660 DEBUG:teuthology.task.console_log:Killing console logger for smithi118
2023-04-27T10:31:26.661 DEBUG:teuthology.exit:Finished running handlers

Another example:
/a/yuriw-2023-04-25_14:52:19-upgrade:octopus-x-pacific-release-distro-default-smithi/7252115/

Related issues 1 (1 open — 0 closed)

Related to Orchestrator - Bug #59529: cluster upgrade stuck with OSDs and MDSs not upgraded.TriagedAdam King

Actions
Actions
Copy link
 #1

Updated by Neha Ojha 12 months ago

  • Assignee set to Adam King

I analyzed /a/yuriw-2023-04-26_20:20:35-upgrade:octopus-x-pacific-release-distro-default-smithi/7255267 and following are my findings.

we upgraded the mons, mgr and one osd at this point

2023-04-27T00:29:20.194 INFO:teuthology.orchestra.run.smithi114.stdout:{
2023-04-27T00:29:20.194 INFO:teuthology.orchestra.run.smithi114.stdout:    "mon": {
2023-04-27T00:29:20.194 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 3
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout:    },
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout:    "mgr": {
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 2
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout:    },
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout:    "osd": {
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 15.2.17-8-gf2877ae3 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable)": 7,
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 1
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout:    },
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout:    "mds": {
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 15.2.17-8-gf2877ae3 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable)": 2
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout:    },
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout:    "overall": {
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 15.2.17-8-gf2877ae3 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable)": 9,
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 6
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout:    }
2023-04-27T00:29:20.197 INFO:teuthology.orchestra.run.smithi114.stdout:}

there was no progress made in more than 10 hours since then and the job eventually died

2023-04-27T10:31:13.047 INFO:teuthology.orchestra.run.smithi114.stdout:{
2023-04-27T10:31:13.048 INFO:teuthology.orchestra.run.smithi114.stdout:    "mon": {
2023-04-27T10:31:13.048 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 3
2023-04-27T10:31:13.048 INFO:teuthology.orchestra.run.smithi114.stdout:    },
2023-04-27T10:31:13.049 INFO:teuthology.orchestra.run.smithi114.stdout:    "mgr": {
2023-04-27T10:31:13.049 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 2
2023-04-27T10:31:13.049 INFO:teuthology.orchestra.run.smithi114.stdout:    },
2023-04-27T10:31:13.049 INFO:teuthology.orchestra.run.smithi114.stdout:    "osd": {
2023-04-27T10:31:13.050 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 15.2.17-8-gf2877ae3 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable)": 7,
2023-04-27T10:31:13.050 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 1
2023-04-27T10:31:13.050 INFO:teuthology.orchestra.run.smithi114.stdout:    },
2023-04-27T10:31:13.050 INFO:teuthology.orchestra.run.smithi114.stdout:    "mds": {
2023-04-27T10:31:13.051 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 15.2.17-8-gf2877ae3 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable)": 2
2023-04-27T10:31:13.051 INFO:teuthology.orchestra.run.smithi114.stdout:    },
2023-04-27T10:31:13.051 INFO:teuthology.orchestra.run.smithi114.stdout:    "overall": {
2023-04-27T10:31:13.051 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 15.2.17-8-gf2877ae3 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable)": 9,
2023-04-27T10:31:13.052 INFO:teuthology.orchestra.run.smithi114.stdout:        "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 6
2023-04-27T10:31:13.052 INFO:teuthology.orchestra.run.smithi114.stdout:    }
2023-04-27T10:31:13.052 INFO:teuthology.orchestra.run.smithi114.stdout:}

osd.0 has been upgraded from the osd's point of view

2023-04-27T00:28:56.673+0000 7faeab280200  0 ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable), process ceph-osd, pid 1
2023-04-27T00:29:00.235+0000 7fae9bf7c700 10 osd.0 246 _collect_metadata {arch=x86_64,back_addr=[v2:172.21.15.114:6804/2182915588,v1:172.21.15.114:6805/2182915588],back_iface=,bluefs=1,bluefs_dedicated_db=0,bluefs_dedicated_wal=0,bluefs_single_shared_device=1,bluestore_bdev_access_mode=blk,bluestore_bdev_block_size=4096,bluestore_bdev_dev_node=/dev/dm-3,bluestore_bdev_devices=nvme0n1,bluestore_bdev_driver=KernelDevice,bluestore_bdev_partition_path=/dev/dm-3,bluestore_bdev_rotational=0,bluestore_bdev_size=95995035648,bluestore_bdev_support_discard=1,bluestore_bdev_type=ssd,ceph_release=pacific,ceph_version=ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable),ceph_version_short=16.2.12-68-g5b35a461,ceph_version_when_created=,container_hostname=smithi114,container_image=quay.ceph.io/ceph-ci/ceph@sha256:4302109dfe042ac15611c85059853b1099a79c00025f707fcf9e8866ab3f408d,cpu=Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz,created_at=,default_device_class=ssd,device_ids=nvme0n1=INTEL_SSDPEDMD400G4_PHFT620400WB400BGN,device_paths=nvme0n1=/dev/disk/by-path/pci-0000:02:00.0-nvme-1,devices=nvme0n1,distro=centos,distro_description=CentOS Stream 8,distro_version=8,front_addr=[v2:172.21.15.114:6802/2182915588,v1:172.21.15.114:6803/2182915588],front_iface=,hb_back_addr=[v2:172.21.15.114:6808/2182915588,v1:172.21.15.114:6809/2182915588],hb_front_addr=[v2:172.21.15.114:6806/2182915588,v1:172.21.15.114:6807/2182915588],hostname=smithi114,journal_rotational=0,kernel_description=#1 SMP Thu Apr 13 17:32:54 UTC 2023,kernel_version=4.18.0-486.el8.x86_64,mem_swap_kb=0,mem_total_kb=32537516,network_numa_unknown_ifaces=back_iface,front_iface,objectstore_numa_node=0,objectstore_numa_nodes=0,os=Linux,osd_data=/var/lib/ceph/osd/ceph-0,osd_objectstore=bluestore,osdspec_affinity=None,rotational=0}

but cephadm says it is starting with unknown version against it

2023-04-27T10:31:12.726 INFO:teuthology.orchestra.run.smithi114.stdout:NAME                      HOST       PORTS   STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION               IMAGE ID      CONTAINER ID
2023-04-27T10:31:12.726 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.a            smithi114          running (10h)    10h ago  10h    24.4M        -  0.20.0                0881eb8f169f  7efa5fc69a42
2023-04-27T10:31:12.726 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.a                 smithi118          running (10h)    10h ago  10h    39.5M        -  6.7.4                 557c83e11646  5a04905b3512
2023-04-27T10:31:12.726 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi114.ilpzfr  smithi114          running (10h)    10h ago  10h    17.4M        -  15.2.17-8-gf2877ae3   96c1dbe86623  ca7236002ee0
2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi118.eanbwj  smithi118          running (10h)    10h ago  10h    14.9M        -  15.2.17-8-gf2877ae3   96c1dbe86623  f4424b8c5453
2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.x                     smithi118  *:8443  running (10h)    10h ago  10h     390M        -  16.2.12-68-g5b35a461  9b232f0f4ceb  299c4dbc08fe
2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.y                     smithi114  *:8443  running (10h)    10h ago  10h     426M        -  16.2.12-68-g5b35a461  9b232f0f4ceb  6e85d942f17e
2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a                     smithi114          running (10h)    10h ago  10h    45.0M    2048M  16.2.12-68-g5b35a461  9b232f0f4ceb  118b6760d3e2
2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b                     smithi118          running (10h)    10h ago  10h    30.2M    2048M  16.2.12-68-g5b35a461  9b232f0f4ceb  e53c2720ead3
2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c                     smithi114          running (10h)    10h ago  10h    29.0M    2048M  16.2.12-68-g5b35a461  9b232f0f4ceb  32ced741e60f
2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.a           smithi114          running (10h)    10h ago  10h    17.8M        -  0.18.1                e5a616e4b9cf  c6636a339a87
2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.b           smithi118          running (10h)    10h ago  10h    17.7M        -  0.18.1                e5a616e4b9cf  a6ccd793215d
2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0                     smithi114          starting               -    -        -    4096M  <unknown>             <unknown>     <unknown>
2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1                     smithi114          running (10h)    10h ago  10h     313M    4096M  15.2.17-8-gf2877ae3   96c1dbe86623  1c2c615fe5ab
2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2                     smithi114          running (10h)    10h ago  10h     233M    4096M  15.2.17-8-gf2877ae3   96c1dbe86623  383c972e3cd6
2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3                     smithi114          running (10h)    10h ago  10h     207M    4096M  15.2.17-8-gf2877ae3   96c1dbe86623  d71730e837ec
2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4                     smithi118          running (10h)    10h ago  10h     296M    4096M  15.2.17-8-gf2877ae3   96c1dbe86623  f8a32fb0b9bf
2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5                     smithi118          running (10h)    10h ago  10h     279M    4096M  15.2.17-8-gf2877ae3   96c1dbe86623  3c4f00fd4a59
2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6                     smithi118          running (10h)    10h ago  10h     180M    4096M  15.2.17-8-gf2877ae3   96c1dbe86623  ca0dd90bf290
2023-04-27T10:31:12.729 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7                     smithi118          running (10h)    10h ago  10h     177M    4096M  15.2.17-8-gf2877ae3   96c1dbe86623  01d207f36912
2023-04-27T10:31:12.729 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.a              smithi118          running (10h)    10h ago  10h    36.9M        -  2.18.1                de242295e225  8a17b1834845

Digging into the cephadm logs, we know that osd.0 was redeployed here

023-04-27 00:28:50,851 7fa53a138b80 INFO Redeploy daemon osd.0 ..

Not sure what happened after this that led the upgrade to get stuck. These error messages stand out.

2023-04-27 00:28:57,627 7fe1cf8d1b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-2ba77aa2-e491-11ed-9b00-001a4aab830c-osd-1" 
2023-04-27 00:28:57,658 7fe1cf8d1b80 DEBUG /bin/podman: 1c2c615fe5ab06917e879b6d9e7db00e417f2a4134040372cf9620d2ab59aea1,quay.ceph.io/ceph-ci/ceph:octopus,96c1dbe866234c8042105c12c4066fb66d4ac9b01c68a704712f8f8f77fc55e8,2023-04-27 00:22:06.827432149 +0000 UTC,

Hey Adam, I am assigning the ticket to you in case you can help make sense of the cephadm logs.

Actions
Copy link
 #2

Updated by Laura Flores 12 months ago

From Adam:

I didn't try any more reruns to find what was causing that particular failure, but I personally think it isn't something that should block the release. It was happening infrequently and I think is something that would be able to be worked around by an actual person if they ran into it.

Actions
Copy link
 #3

Updated by Laura Flores 12 months ago

Seeing something similar happen in this job.

/a/yuriw-2023-04-26_01:16:19-rados-wip-yuri11-testing-2023-04-25-1605-pacific-distro-default-smithi/7253983

Description: rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final}

The ceph versions haven't been refreshed in 10 hours. osd.1 has an "unknown" version.

2023-04-30T00:07:41.046 INFO:teuthology.orchestra.run.smithi149.stdout:NAME                          HOST       PORTS        STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION               IMAGE ID      CONTAINER ID
2023-04-30T00:07:41.047 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.smithi149        smithi149  *:9093,9094  running (10h)    10h ago  10h    24.4M        -  0.20.0                0881eb8f169f  881dd3a04784
2023-04-30T00:07:41.047 INFO:teuthology.orchestra.run.smithi149.stdout:crash.smithi149               smithi149               running (10h)    10h ago  10h    7306k        -  16.2.12-89-g8d175760  21fa24760dad  32a04af059a3
2023-04-30T00:07:41.047 INFO:teuthology.orchestra.run.smithi149.stdout:crash.smithi170               smithi170               running (10h)    10h ago  10h    7285k        -  16.2.12-89-g8d175760  21fa24760dad  192e56ded396
2023-04-30T00:07:41.047 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.smithi149             smithi149  *:3000       running (10h)    10h ago  10h    37.7M        -  6.7.4                 557c83e11646  7eeac5669879
2023-04-30T00:07:41.047 INFO:teuthology.orchestra.run.smithi149.stdout:mds.foofs.smithi149.ihtlve    smithi149               running (10h)    10h ago  10h    13.5M        -  16.2.4                8d91d370c2b8  a3f9cbdaf877
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:mds.foofs.smithi170.flglis    smithi170               running (10h)    10h ago  10h    31.8M        -  16.2.4                8d91d370c2b8  3cac984b365a
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.smithi149.hlsuma          smithi149  *:8443,9283  running (10h)    10h ago  10h     442M        -  16.2.12-89-g8d175760  21fa24760dad  1c2cf34729ce
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.smithi170.dgqyog          smithi170  *:8443,9283  running (10h)    10h ago  10h     394M        -  16.2.12-89-g8d175760  21fa24760dad  5bc06f84c7b2
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:mon.smithi149                 smithi149               running (10h)    10h ago  10h    69.5M    2048M  16.2.12-89-g8d175760  21fa24760dad  f08bfd12e9ca
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:mon.smithi170                 smithi170               running (10h)    10h ago  10h    57.8M    2048M  16.2.12-89-g8d175760  21fa24760dad  1599b9c61da2
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:nfs.foo.0.0.smithi149.ubgmyv  smithi149  *:2049       running (10h)    10h ago  10h    59.0M        -  3.5                   8d91d370c2b8  d70f2a22d0c4
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:nfs.foo.1.0.smithi170.mxnhbi  smithi170  *:2049       running (10h)    10h ago  10h    79.5M        -  3.5                   8d91d370c2b8  d532c3b84174
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.smithi149       smithi149  *:9100       running (10h)    10h ago  10h    18.2M        -  0.18.1                e5a616e4b9cf  884ae1e0ada8
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.smithi170       smithi170  *:9100       running (10h)    10h ago  10h    18.4M        -  0.18.1                e5a616e4b9cf  c9cba6b325fc
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0                         smithi149               running (10h)    10h ago  10h    11.8M    4096M  16.2.12-89-g8d175760  21fa24760dad  ac53d807764a
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1                         smithi149               starting               -    -        -    4096M  <unknown>             <unknown>     <unknown>
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2                         smithi149               running (10h)    10h ago  10h     752M    4096M  16.2.4                8d91d370c2b8  b2bee618e6b9
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3                         smithi149               running (10h)    10h ago  10h    1063M    4096M  16.2.4                8d91d370c2b8  b0771b222e56
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4                         smithi170               running (10h)    10h ago  10h    1057M    4096M  16.2.4                8d91d370c2b8  cc966c9bb5da
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5                         smithi170               running (10h)    10h ago  10h    1311M    4096M  16.2.4                8d91d370c2b8  a7a195701819
2023-04-30T00:07:41.050 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6                         smithi170               running (10h)    10h ago  10h    1043M    4096M  16.2.4                8d91d370c2b8  52728e01b84e
2023-04-30T00:07:41.050 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7                         smithi170               running (10h)    10h ago  10h    1554M    4096M  16.2.4                8d91d370c2b8  02ca6e002ecc
2023-04-30T00:07:41.050 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.smithi149          smithi149  *:9095       running (10h)    10h ago  10h    54.3M        -  2.18.1                de242295e225  b7616dda8f28

However, in the osd log, the osd is at version 16.2.12-89-g8d175760. The only difference between this and the first example is that the osd has not been upgraded yet from the osd's point of view.

2023-04-29T13:21:04.019+0000 7f88987fa700 10 osd.1 77 _collect_metadata {arch=x86_64,back_addr=[v2:172.21.15.149:6812/3248254058,v1:172.21.15.149:6813/3248254058],back_iface=,bluefs=1,bluefs_dedicated_db=0,bluefs_dedicated_wal=0,bluefs_single_shared_device=1,bluestore_bdev_access_mode=blk,bluestore_bdev_block_size=4096,bluestore_bdev_dev_node=/dev/dm-2,bluestore_bdev_devices=nvme0n1,bluestore_bdev_driver=KernelDevice,bluestore_bdev_partition_path=/dev/dm-2,bluestore_bdev_rotational=0,bluestore_bdev_size=95995035648,bluestore_bdev_support_discard=1,bluestore_bdev_type=ssd,ceph_release=pacific,ceph_version=ceph version 16.2.12-89-g8d175760 (8d17576050e846ecd4a9899bc7d8ebbf771b4de8) pacific (stable),ceph_version_short=16.2.12-89-g8d175760,ceph_version_when_created=,container_hostname=smithi149,container_image=quay.ceph.io/ceph-ci/ceph@sha256:7a74efaa3885d5857fc7f29eaab95c6144dfebca3e531340a2f32f46afbb8d8d,cpu=Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz,created_at=,default_device_class=ssd,device_ids=nvme0n1=INTEL_SSDPEDMD400G4_CVFT623300CK400BGN,device_paths=nvme0n1=/dev/disk/by-path/pci-0000:02:00.0-nvme-1,devices=nvme0n1,distro=centos,distro_description=CentOS Stream 8,distro_version=8,front_addr=[v2:172.21.15.149:6810/3248254058,v1:172.21.15.149:6811/3248254058],front_iface=,hb_back_addr=[v2:172.21.15.149:6816/3248254058,v1:172.21.15.149:6817/3248254058],hb_front_addr=[v2:172.21.15.149:6814/3248254058,v1:172.21.15.149:6815/3248254058],hostname=smithi149,journal_rotational=0,kernel_description=#1 SMP Fri Apr 21 18:01:53 UTC 2023,kernel_version=4.18.0-488.el8.x86_64,mem_swap_kb=0,mem_total_kb=16022544,network_numa_unknown_ifaces=back_iface,front_iface,objectstore_numa_node=0,objectstore_numa_nodes=0,os=Linux,osd_data=/var/lib/ceph/osd/ceph-1,osd_objectstore=bluestore,osdspec_affinity=None,rotational=0}

Cephadm can't find several objects:

2023-04-29 13:19:06,586 7f1001b88b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-574c31b6-e68f-11ed-9b00-001a4aab830c-osd-0" 
2023-04-29 13:19:06,622 7f1001b88b80 DEBUG /bin/podman: 0ffbed78faff21db20ad23cdbf573726e357f0e415659800b8c3c9eb72549a93,docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29,2023-04-29 13:14:16.838408371 +0000 UTC,
2023-04-29 13:19:06,632 7f1001b88b80 DEBUG systemctl: enabled
2023-04-29 13:19:06,639 7f1001b88b80 DEBUG systemctl: active
2023-04-29 13:19:06,732 7f1001b88b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-574c31b6-e68f-11ed-9b00-001a4aab830c-osd-1" 
2023-04-29 13:19:06,768 7f1001b88b80 DEBUG /bin/podman: 1ee659078e94f0b2590a8987df393794602b40ee462371e544b4b2b9e5956be5,docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29,2023-04-29 13:14:33.094537011 +0000 UTC,
2023-04-29 13:19:06,777 7f1001b88b80 DEBUG systemctl: enabled
2023-04-29 13:19:06,785 7f1001b88b80 DEBUG systemctl: active
2023-04-29 13:19:06,886 7f1001b88b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-574c31b6-e68f-11ed-9b00-001a4aab830c-osd-2" 
2023-04-29 13:19:06,922 7f1001b88b80 DEBUG /bin/podman: b2bee618e6b9291e44bf47cb85f6aa50a8a5f9d4a6edcd7098256700065c3b52,docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29,2023-04-29 13:14:48.076020763 +0000 UTC,
2023-04-29 13:19:06,932 7f1001b88b80 DEBUG systemctl: enabled
2023-04-29 13:19:06,938 7f1001b88b80 DEBUG systemctl: active
2023-04-29 13:19:07,036 7f1001b88b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-574c31b6-e68f-11ed-9b00-001a4aab830c-osd-3" 
2023-04-29 13:19:07,073 7f1001b88b80 DEBUG /bin/podman: b0771b222e567405930fe34e1bba84e522a5be7229ff12692deffa483c024f12,docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29,2023-04-29 13:15:03.831303683 +0000 UTC,
2023-04-29 13:19:07,082 7f1001b88b80 DEBUG systemctl: enabled
2023-04-29 13:19:07,089 7f1001b88b80 DEBUG systemctl: active
2023-04-29 13:19:07,182 7f1001b88b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-574c31b6-e68f-11ed-9b00-001a4aab830c-mds-foofs-smithi149-ihtlve" 
2023-04-29 13:19:07,216 7f1001b88b80 DEBUG /bin/podman: a3f9cbdaf877c9799eaf8e81bd841e92ce868b3bc9e06f3781f213a85f2231eb,docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29,2023-04-29 13:17:15.060599059 +0000 UTC,
2023-04-29 13:19:07,227 7f1001b88b80 DEBUG systemctl: enabled
2023-04-29 13:19:07,234 7f1001b88b80 DEBUG systemctl: active
2023-04-29 13:19:07,332 7f1001b88b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-574c31b6-e68f-11ed-9b00-001a4aab830c-nfs-foo-smithi149" 
2023-04-29 13:19:07,366 7f1001b88b80 DEBUG /bin/podman: 23c175eefac5c3522b41cc03bbf38550a4ca446966f42c1828975e61c9605824,docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,8d91d370c2b86c07de46146ab

Actions
Copy link
 #4

Updated by Laura Flores 12 months ago

  • Subject changed from upgrade:octopus-x (pacific): StopSignal SIGTERM failed to stop container ceph-2ba77aa2-e491-11ed-9b00-001a4aab830c-mgr.x in 10 seconds, resorting to SIGKILL to upgrade: unkown ceph version causes upgrade to get stuck
Actions
Copy link
 #5

Updated by Laura Flores 12 months ago

  • Backport set to pacific
Actions
Copy link
 #6

Updated by Laura Flores 12 months ago

  • Project changed from Ceph to Orchestrator
Actions
Copy link
 #7

Updated by Laura Flores 10 months ago

  • Related to Bug #59529: cluster upgrade stuck with OSDs and MDSs not upgraded. added
Actions
Copy link

Also available in: Atom PDF