Actions
Bug #59604
openupgrade: unkown ceph version causes upgrade to get stuck
Added by Laura Flores about 1 year ago. Updated about 1 year ago.
% Done:
0%
Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
/a/yuriw-2023-04-26_20:20:35-upgrade:octopus-x-pacific-release-distro-default-smithi/7255267/
2023-04-27T00:27:52.980 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Apr 27 00:27:52 smithi118 bash[149104]: time="2023-04-27T00:27:52Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-2ba77aa2-e491-11ed-9b00-001a4aab830c-mgr.x in 10 seconds, resorting to SIGKILL"
...
2023-04-27T10:31:26.597 DEBUG:teuthology.exit:Got signal 15; running 1 handler...
2023-04-27T10:31:26.626 DEBUG:teuthology.task.console_log:Killing console logger for smithi114
2023-04-27T10:31:26.660 DEBUG:teuthology.task.console_log:Killing console logger for smithi118
2023-04-27T10:31:26.661 DEBUG:teuthology.exit:Finished running handlers
Another example:
/a/yuriw-2023-04-25_14:52:19-upgrade:octopus-x-pacific-release-distro-default-smithi/7252115/
Updated by Neha Ojha about 1 year ago
- Assignee set to Adam King
I analyzed /a/yuriw-2023-04-26_20:20:35-upgrade:octopus-x-pacific-release-distro-default-smithi/7255267 and following are my findings.
we upgraded the mons, mgr and one osd at this point
2023-04-27T00:29:20.194 INFO:teuthology.orchestra.run.smithi114.stdout:{
2023-04-27T00:29:20.194 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": {
2023-04-27T00:29:20.194 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 3
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout: },
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": {
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 2
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout: },
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": {
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 15.2.17-8-gf2877ae3 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable)": 7,
2023-04-27T00:29:20.195 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 1
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout: },
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": {
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 15.2.17-8-gf2877ae3 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable)": 2
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout: },
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": {
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 15.2.17-8-gf2877ae3 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable)": 9,
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 6
2023-04-27T00:29:20.196 INFO:teuthology.orchestra.run.smithi114.stdout: }
2023-04-27T00:29:20.197 INFO:teuthology.orchestra.run.smithi114.stdout:}
there was no progress made in more than 10 hours since then and the job eventually died
2023-04-27T10:31:13.047 INFO:teuthology.orchestra.run.smithi114.stdout:{
2023-04-27T10:31:13.048 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": {
2023-04-27T10:31:13.048 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 3
2023-04-27T10:31:13.048 INFO:teuthology.orchestra.run.smithi114.stdout: },
2023-04-27T10:31:13.049 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": {
2023-04-27T10:31:13.049 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 2
2023-04-27T10:31:13.049 INFO:teuthology.orchestra.run.smithi114.stdout: },
2023-04-27T10:31:13.049 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": {
2023-04-27T10:31:13.050 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 15.2.17-8-gf2877ae3 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable)": 7,
2023-04-27T10:31:13.050 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 1
2023-04-27T10:31:13.050 INFO:teuthology.orchestra.run.smithi114.stdout: },
2023-04-27T10:31:13.050 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": {
2023-04-27T10:31:13.051 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 15.2.17-8-gf2877ae3 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable)": 2
2023-04-27T10:31:13.051 INFO:teuthology.orchestra.run.smithi114.stdout: },
2023-04-27T10:31:13.051 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": {
2023-04-27T10:31:13.051 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 15.2.17-8-gf2877ae3 (f2877ae32a72fc25acadef57597f44988b805c38) octopus (stable)": 9,
2023-04-27T10:31:13.052 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable)": 6
2023-04-27T10:31:13.052 INFO:teuthology.orchestra.run.smithi114.stdout: }
2023-04-27T10:31:13.052 INFO:teuthology.orchestra.run.smithi114.stdout:}
osd.0 has been upgraded from the osd's point of view
2023-04-27T00:28:56.673+0000 7faeab280200 0 ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable), process ceph-osd, pid 1
2023-04-27T00:29:00.235+0000 7fae9bf7c700 10 osd.0 246 _collect_metadata {arch=x86_64,back_addr=[v2:172.21.15.114:6804/2182915588,v1:172.21.15.114:6805/2182915588],back_iface=,bluefs=1,bluefs_dedicated_db=0,bluefs_dedicated_wal=0,bluefs_single_shared_device=1,bluestore_bdev_access_mode=blk,bluestore_bdev_block_size=4096,bluestore_bdev_dev_node=/dev/dm-3,bluestore_bdev_devices=nvme0n1,bluestore_bdev_driver=KernelDevice,bluestore_bdev_partition_path=/dev/dm-3,bluestore_bdev_rotational=0,bluestore_bdev_size=95995035648,bluestore_bdev_support_discard=1,bluestore_bdev_type=ssd,ceph_release=pacific,ceph_version=ceph version 16.2.12-68-g5b35a461 (5b35a461e1c1b935b2b3fc7c43d68a58c1a41547) pacific (stable),ceph_version_short=16.2.12-68-g5b35a461,ceph_version_when_created=,container_hostname=smithi114,container_image=quay.ceph.io/ceph-ci/ceph@sha256:4302109dfe042ac15611c85059853b1099a79c00025f707fcf9e8866ab3f408d,cpu=Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz,created_at=,default_device_class=ssd,device_ids=nvme0n1=INTEL_SSDPEDMD400G4_PHFT620400WB400BGN,device_paths=nvme0n1=/dev/disk/by-path/pci-0000:02:00.0-nvme-1,devices=nvme0n1,distro=centos,distro_description=CentOS Stream 8,distro_version=8,front_addr=[v2:172.21.15.114:6802/2182915588,v1:172.21.15.114:6803/2182915588],front_iface=,hb_back_addr=[v2:172.21.15.114:6808/2182915588,v1:172.21.15.114:6809/2182915588],hb_front_addr=[v2:172.21.15.114:6806/2182915588,v1:172.21.15.114:6807/2182915588],hostname=smithi114,journal_rotational=0,kernel_description=#1 SMP Thu Apr 13 17:32:54 UTC 2023,kernel_version=4.18.0-486.el8.x86_64,mem_swap_kb=0,mem_total_kb=32537516,network_numa_unknown_ifaces=back_iface,front_iface,objectstore_numa_node=0,objectstore_numa_nodes=0,os=Linux,osd_data=/var/lib/ceph/osd/ceph-0,osd_objectstore=bluestore,osdspec_affinity=None,rotational=0}
but cephadm says it is starting with unknown version against it
2023-04-27T10:31:12.726 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-04-27T10:31:12.726 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.a smithi114 running (10h) 10h ago 10h 24.4M - 0.20.0 0881eb8f169f 7efa5fc69a42 2023-04-27T10:31:12.726 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.a smithi118 running (10h) 10h ago 10h 39.5M - 6.7.4 557c83e11646 5a04905b3512 2023-04-27T10:31:12.726 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi114.ilpzfr smithi114 running (10h) 10h ago 10h 17.4M - 15.2.17-8-gf2877ae3 96c1dbe86623 ca7236002ee0 2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi118.eanbwj smithi118 running (10h) 10h ago 10h 14.9M - 15.2.17-8-gf2877ae3 96c1dbe86623 f4424b8c5453 2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.x smithi118 *:8443 running (10h) 10h ago 10h 390M - 16.2.12-68-g5b35a461 9b232f0f4ceb 299c4dbc08fe 2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.y smithi114 *:8443 running (10h) 10h ago 10h 426M - 16.2.12-68-g5b35a461 9b232f0f4ceb 6e85d942f17e 2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a smithi114 running (10h) 10h ago 10h 45.0M 2048M 16.2.12-68-g5b35a461 9b232f0f4ceb 118b6760d3e2 2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b smithi118 running (10h) 10h ago 10h 30.2M 2048M 16.2.12-68-g5b35a461 9b232f0f4ceb e53c2720ead3 2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c smithi114 running (10h) 10h ago 10h 29.0M 2048M 16.2.12-68-g5b35a461 9b232f0f4ceb 32ced741e60f 2023-04-27T10:31:12.727 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.a smithi114 running (10h) 10h ago 10h 17.8M - 0.18.1 e5a616e4b9cf c6636a339a87 2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.b smithi118 running (10h) 10h ago 10h 17.7M - 0.18.1 e5a616e4b9cf a6ccd793215d 2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 starting - - - 4096M <unknown> <unknown> <unknown> 2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (10h) 10h ago 10h 313M 4096M 15.2.17-8-gf2877ae3 96c1dbe86623 1c2c615fe5ab 2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (10h) 10h ago 10h 233M 4096M 15.2.17-8-gf2877ae3 96c1dbe86623 383c972e3cd6 2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi114 running (10h) 10h ago 10h 207M 4096M 15.2.17-8-gf2877ae3 96c1dbe86623 d71730e837ec 2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi118 running (10h) 10h ago 10h 296M 4096M 15.2.17-8-gf2877ae3 96c1dbe86623 f8a32fb0b9bf 2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi118 running (10h) 10h ago 10h 279M 4096M 15.2.17-8-gf2877ae3 96c1dbe86623 3c4f00fd4a59 2023-04-27T10:31:12.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 smithi118 running (10h) 10h ago 10h 180M 4096M 15.2.17-8-gf2877ae3 96c1dbe86623 ca0dd90bf290 2023-04-27T10:31:12.729 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 smithi118 running (10h) 10h ago 10h 177M 4096M 15.2.17-8-gf2877ae3 96c1dbe86623 01d207f36912 2023-04-27T10:31:12.729 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.a smithi118 running (10h) 10h ago 10h 36.9M - 2.18.1 de242295e225 8a17b1834845
Digging into the cephadm logs, we know that osd.0 was redeployed here
023-04-27 00:28:50,851 7fa53a138b80 INFO Redeploy daemon osd.0 ..
Not sure what happened after this that led the upgrade to get stuck. These error messages stand out.
2023-04-27 00:28:57,627 7fe1cf8d1b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-2ba77aa2-e491-11ed-9b00-001a4aab830c-osd-1" 2023-04-27 00:28:57,658 7fe1cf8d1b80 DEBUG /bin/podman: 1c2c615fe5ab06917e879b6d9e7db00e417f2a4134040372cf9620d2ab59aea1,quay.ceph.io/ceph-ci/ceph:octopus,96c1dbe866234c8042105c12c4066fb66d4ac9b01c68a704712f8f8f77fc55e8,2023-04-27 00:22:06.827432149 +0000 UTC,
Hey Adam, I am assigning the ticket to you in case you can help make sense of the cephadm logs.
Updated by Laura Flores about 1 year ago
From Adam:
I didn't try any more reruns to find what was causing that particular failure, but I personally think it isn't something that should block the release. It was happening infrequently and I think is something that would be able to be worked around by an actual person if they ran into it.
Updated by Laura Flores about 1 year ago
Seeing something similar happen in this job.
/a/yuriw-2023-04-26_01:16:19-rados-wip-yuri11-testing-2023-04-25-1605-pacific-distro-default-smithi/7253983
Description: rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final}
The ceph versions haven't been refreshed in 10 hours. osd.1 has an "unknown" version.
2023-04-30T00:07:41.046 INFO:teuthology.orchestra.run.smithi149.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
2023-04-30T00:07:41.047 INFO:teuthology.orchestra.run.smithi149.stdout:alertmanager.smithi149 smithi149 *:9093,9094 running (10h) 10h ago 10h 24.4M - 0.20.0 0881eb8f169f 881dd3a04784
2023-04-30T00:07:41.047 INFO:teuthology.orchestra.run.smithi149.stdout:crash.smithi149 smithi149 running (10h) 10h ago 10h 7306k - 16.2.12-89-g8d175760 21fa24760dad 32a04af059a3
2023-04-30T00:07:41.047 INFO:teuthology.orchestra.run.smithi149.stdout:crash.smithi170 smithi170 running (10h) 10h ago 10h 7285k - 16.2.12-89-g8d175760 21fa24760dad 192e56ded396
2023-04-30T00:07:41.047 INFO:teuthology.orchestra.run.smithi149.stdout:grafana.smithi149 smithi149 *:3000 running (10h) 10h ago 10h 37.7M - 6.7.4 557c83e11646 7eeac5669879
2023-04-30T00:07:41.047 INFO:teuthology.orchestra.run.smithi149.stdout:mds.foofs.smithi149.ihtlve smithi149 running (10h) 10h ago 10h 13.5M - 16.2.4 8d91d370c2b8 a3f9cbdaf877
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:mds.foofs.smithi170.flglis smithi170 running (10h) 10h ago 10h 31.8M - 16.2.4 8d91d370c2b8 3cac984b365a
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.smithi149.hlsuma smithi149 *:8443,9283 running (10h) 10h ago 10h 442M - 16.2.12-89-g8d175760 21fa24760dad 1c2cf34729ce
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:mgr.smithi170.dgqyog smithi170 *:8443,9283 running (10h) 10h ago 10h 394M - 16.2.12-89-g8d175760 21fa24760dad 5bc06f84c7b2
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:mon.smithi149 smithi149 running (10h) 10h ago 10h 69.5M 2048M 16.2.12-89-g8d175760 21fa24760dad f08bfd12e9ca
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:mon.smithi170 smithi170 running (10h) 10h ago 10h 57.8M 2048M 16.2.12-89-g8d175760 21fa24760dad 1599b9c61da2
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:nfs.foo.0.0.smithi149.ubgmyv smithi149 *:2049 running (10h) 10h ago 10h 59.0M - 3.5 8d91d370c2b8 d70f2a22d0c4
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:nfs.foo.1.0.smithi170.mxnhbi smithi170 *:2049 running (10h) 10h ago 10h 79.5M - 3.5 8d91d370c2b8 d532c3b84174
2023-04-30T00:07:41.048 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.smithi149 smithi149 *:9100 running (10h) 10h ago 10h 18.2M - 0.18.1 e5a616e4b9cf 884ae1e0ada8
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:node-exporter.smithi170 smithi170 *:9100 running (10h) 10h ago 10h 18.4M - 0.18.1 e5a616e4b9cf c9cba6b325fc
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:osd.0 smithi149 running (10h) 10h ago 10h 11.8M 4096M 16.2.12-89-g8d175760 21fa24760dad ac53d807764a
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:osd.1 smithi149 starting - - - 4096M <unknown> <unknown> <unknown>
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:osd.2 smithi149 running (10h) 10h ago 10h 752M 4096M 16.2.4 8d91d370c2b8 b2bee618e6b9
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:osd.3 smithi149 running (10h) 10h ago 10h 1063M 4096M 16.2.4 8d91d370c2b8 b0771b222e56
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:osd.4 smithi170 running (10h) 10h ago 10h 1057M 4096M 16.2.4 8d91d370c2b8 cc966c9bb5da
2023-04-30T00:07:41.049 INFO:teuthology.orchestra.run.smithi149.stdout:osd.5 smithi170 running (10h) 10h ago 10h 1311M 4096M 16.2.4 8d91d370c2b8 a7a195701819
2023-04-30T00:07:41.050 INFO:teuthology.orchestra.run.smithi149.stdout:osd.6 smithi170 running (10h) 10h ago 10h 1043M 4096M 16.2.4 8d91d370c2b8 52728e01b84e
2023-04-30T00:07:41.050 INFO:teuthology.orchestra.run.smithi149.stdout:osd.7 smithi170 running (10h) 10h ago 10h 1554M 4096M 16.2.4 8d91d370c2b8 02ca6e002ecc
2023-04-30T00:07:41.050 INFO:teuthology.orchestra.run.smithi149.stdout:prometheus.smithi149 smithi149 *:9095 running (10h) 10h ago 10h 54.3M - 2.18.1 de242295e225 b7616dda8f28
However, in the osd log, the osd is at version 16.2.12-89-g8d175760. The only difference between this and the first example is that the osd has not been upgraded yet from the osd's point of view.
2023-04-29T13:21:04.019+0000 7f88987fa700 10 osd.1 77 _collect_metadata {arch=x86_64,back_addr=[v2:172.21.15.149:6812/3248254058,v1:172.21.15.149:6813/3248254058],back_iface=,bluefs=1,bluefs_dedicated_db=0,bluefs_dedicated_wal=0,bluefs_single_shared_device=1,bluestore_bdev_access_mode=blk,bluestore_bdev_block_size=4096,bluestore_bdev_dev_node=/dev/dm-2,bluestore_bdev_devices=nvme0n1,bluestore_bdev_driver=KernelDevice,bluestore_bdev_partition_path=/dev/dm-2,bluestore_bdev_rotational=0,bluestore_bdev_size=95995035648,bluestore_bdev_support_discard=1,bluestore_bdev_type=ssd,ceph_release=pacific,ceph_version=ceph version 16.2.12-89-g8d175760 (8d17576050e846ecd4a9899bc7d8ebbf771b4de8) pacific (stable),ceph_version_short=16.2.12-89-g8d175760,ceph_version_when_created=,container_hostname=smithi149,container_image=quay.ceph.io/ceph-ci/ceph@sha256:7a74efaa3885d5857fc7f29eaab95c6144dfebca3e531340a2f32f46afbb8d8d,cpu=Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz,created_at=,default_device_class=ssd,device_ids=nvme0n1=INTEL_SSDPEDMD400G4_CVFT623300CK400BGN,device_paths=nvme0n1=/dev/disk/by-path/pci-0000:02:00.0-nvme-1,devices=nvme0n1,distro=centos,distro_description=CentOS Stream 8,distro_version=8,front_addr=[v2:172.21.15.149:6810/3248254058,v1:172.21.15.149:6811/3248254058],front_iface=,hb_back_addr=[v2:172.21.15.149:6816/3248254058,v1:172.21.15.149:6817/3248254058],hb_front_addr=[v2:172.21.15.149:6814/3248254058,v1:172.21.15.149:6815/3248254058],hostname=smithi149,journal_rotational=0,kernel_description=#1 SMP Fri Apr 21 18:01:53 UTC 2023,kernel_version=4.18.0-488.el8.x86_64,mem_swap_kb=0,mem_total_kb=16022544,network_numa_unknown_ifaces=back_iface,front_iface,objectstore_numa_node=0,objectstore_numa_nodes=0,os=Linux,osd_data=/var/lib/ceph/osd/ceph-1,osd_objectstore=bluestore,osdspec_affinity=None,rotational=0}
Cephadm can't find several objects:
2023-04-29 13:19:06,586 7f1001b88b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-574c31b6-e68f-11ed-9b00-001a4aab830c-osd-0"
2023-04-29 13:19:06,622 7f1001b88b80 DEBUG /bin/podman: 0ffbed78faff21db20ad23cdbf573726e357f0e415659800b8c3c9eb72549a93,docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29,2023-04-29 13:14:16.838408371 +0000 UTC,
2023-04-29 13:19:06,632 7f1001b88b80 DEBUG systemctl: enabled
2023-04-29 13:19:06,639 7f1001b88b80 DEBUG systemctl: active
2023-04-29 13:19:06,732 7f1001b88b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-574c31b6-e68f-11ed-9b00-001a4aab830c-osd-1"
2023-04-29 13:19:06,768 7f1001b88b80 DEBUG /bin/podman: 1ee659078e94f0b2590a8987df393794602b40ee462371e544b4b2b9e5956be5,docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29,2023-04-29 13:14:33.094537011 +0000 UTC,
2023-04-29 13:19:06,777 7f1001b88b80 DEBUG systemctl: enabled
2023-04-29 13:19:06,785 7f1001b88b80 DEBUG systemctl: active
2023-04-29 13:19:06,886 7f1001b88b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-574c31b6-e68f-11ed-9b00-001a4aab830c-osd-2"
2023-04-29 13:19:06,922 7f1001b88b80 DEBUG /bin/podman: b2bee618e6b9291e44bf47cb85f6aa50a8a5f9d4a6edcd7098256700065c3b52,docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29,2023-04-29 13:14:48.076020763 +0000 UTC,
2023-04-29 13:19:06,932 7f1001b88b80 DEBUG systemctl: enabled
2023-04-29 13:19:06,938 7f1001b88b80 DEBUG systemctl: active
2023-04-29 13:19:07,036 7f1001b88b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-574c31b6-e68f-11ed-9b00-001a4aab830c-osd-3"
2023-04-29 13:19:07,073 7f1001b88b80 DEBUG /bin/podman: b0771b222e567405930fe34e1bba84e522a5be7229ff12692deffa483c024f12,docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29,2023-04-29 13:15:03.831303683 +0000 UTC,
2023-04-29 13:19:07,082 7f1001b88b80 DEBUG systemctl: enabled
2023-04-29 13:19:07,089 7f1001b88b80 DEBUG systemctl: active
2023-04-29 13:19:07,182 7f1001b88b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-574c31b6-e68f-11ed-9b00-001a4aab830c-mds-foofs-smithi149-ihtlve"
2023-04-29 13:19:07,216 7f1001b88b80 DEBUG /bin/podman: a3f9cbdaf877c9799eaf8e81bd841e92ce868b3bc9e06f3781f213a85f2231eb,docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29,2023-04-29 13:17:15.060599059 +0000 UTC,
2023-04-29 13:19:07,227 7f1001b88b80 DEBUG systemctl: enabled
2023-04-29 13:19:07,234 7f1001b88b80 DEBUG systemctl: active
2023-04-29 13:19:07,332 7f1001b88b80 DEBUG /bin/podman: Error: inspecting object: no such object: "ceph-574c31b6-e68f-11ed-9b00-001a4aab830c-nfs-foo-smithi149"
2023-04-29 13:19:07,366 7f1001b88b80 DEBUG /bin/podman: 23c175eefac5c3522b41cc03bbf38550a4ca446966f42c1828975e61c9605824,docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949,8d91d370c2b86c07de46146ab
Updated by Laura Flores about 1 year ago
- Subject changed from upgrade:octopus-x (pacific): StopSignal SIGTERM failed to stop container ceph-2ba77aa2-e491-11ed-9b00-001a4aab830c-mgr.x in 10 seconds, resorting to SIGKILL to upgrade: unkown ceph version causes upgrade to get stuck
Updated by Laura Flores about 1 year ago
- Project changed from Ceph to Orchestrator
Updated by Laura Flores 11 months ago
- Related to Bug #59529: cluster upgrade stuck with OSDs and MDSs not upgraded. added
Actions