Ceph » Orchestrator

2021-02-25T18:53:57.786 INFO:teuthology.orchestra.run.smithi141.stderr:Non-zero exit code 127 from /bin/podman run --rm --ipc=host --net=host --entrypoint stat -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:cf3694ad8a53ad63a49b370c47fe0396f994a744 -e NODE_NAME=smithi141 quay.ceph.io/ceph-ci/ceph:cf3694ad8a53ad63a49b370c47fe0396f994a744 -c %u %g /var/lib/ceph
2021-02-25T18:53:57.795 INFO:teuthology.orchestra.run.smithi141.stderr:stat: stderr Error: OCI runtime error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: Unit libpod-224e2eaa6207f5d6942dea5588c93063988b2ea02d99398af2f439ce823cf1a0.scope not found.

/a/sage-2021-03-01_20:25:17-rados-wip-sage4-testing-2021-03-01-1042-distro-basic-gibba/5924180

description: rados/cephadm/with-work/{distro/ubuntu_20.04_podman_testing fixed-2 mode/packaged
mon_election/classic msgr/async-v1only start tasks/rados_python}

/ceph/teuthology-archive/yuriw-2021-03-24_23:05:33-rados-wip-yuri2-testing-2021-03-24-1212-octopus-distro-basic-smithi/5995730/teuthology.log

2021-03-27T10:19:01.795 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:bc61c498949fff8c3da535d41cd130d28d16cdc3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d65ef44-8ee5-11eb-ab2c-001a4aab830c -- ceph orch daemon add mon smithi148:172.21.15.148=smithi148
2021-03-27T10:19:04.634 INFO:teuthology.orchestra.run.smithi148.stderr:Error: OCI runtime error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: Unit libpod-6400e71e1d28e76e078f9e2a63b55433e3c4bc8d5df6b6bb0e2076c30605eb6b.scope not found.
2021-03-27T10:19:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 127

https://pulpito.ceph.com/pdonnell-2021-07-12_18:17:18-fs:workload-wip-pdonnell-testing-20210710.030422-distro-basic-smithi/6266629/

fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 

https://bugzilla.redhat.com/show_bug.cgi?id=1972209

https://github.com/ceph/ceph/pull/42000#issuecomment-879313562

http://qa-proxy.ceph.com/teuthology/ideepika-2021-08-29_11:15:03-rados-wip-deepika2-testing-2021-08-27-0331-distro-basic-smithi/6365531/teuthology.log

Deepika, I don't think we can do anything about this at this time.

/a/yuriw-2021-09-16_18:23:18-rados-wip-yuri2-testing-2021-09-16-0923-distro-basic-smithi/6393507

http://qa-proxy.ceph.com/teuthology/ideepika-2021-09-27_14:52:49-rados-wip-deepika-testing-2021-09-27-1601-distro-basic-smithi/6409132/teuthology.log

https://bugzilla.redhat.com/show_bug.cgi?id=2019335 <- for centos stream

/a/yuriw-2021-12-03_15:27:18-rados-wip-yuri11-testing-2021-12-02-1451-distro-default-smithi/6542708

/a/yuriw-2021-12-09_00:18:57-rados-wip-yuri-testing-2021-12-08-1336-distro-default-smithi/6553774/

/a/lflores-2022-01-05_19:04:35-rados-wip-lflores-mgr-rocksdb-distro-default-smithi/6596855

/a/yuriw-2022-01-08_17:57:43-rados-wip-yuri8-testing-2022-01-07-1541-distro-default-smithi/6603250

/a/yuriw-2022-02-17_22:49:55-rados-wip-yuri3-testing-2022-02-17-1256-distro-default-smithi/6692382

/a/yuriw-2022-02-17_23:23:56-rados-wip-yuri7-testing-2022-02-17-0852-pacific-distro-default-smithi/6692693

/a/yuriw-2022-03-17_14:54:32-rados-wip-yuri10-testing-2022-03-16-1432-pacific-distro-default-smithi/6742149

Reading the BZ thread it seems that:
1- This issue is caused by a BUG in runc (https://github.com/opencontainers/runc/pull/2614) which should be solved in runc version >= 1.0.1
2- backport of the fix to container-tools 3.0 in RHEL 8.4 is tracked by https://bugzilla.redhat.com/show_bug.cgi?id=2000570

My doubt is: which versions of these packages are being installed on the teuthology machines where these tests are failed?

Can you please check which runtime is being used by podman by running:

podman info --debug

and

dnf info runc

Hi Redouane,

as we have discussed offline,

smithi161 which was one of the machines used in the job that came across this bug:
/a/yuriw-2022-03-17_14:54:32-rados-wip-yuri10-testing-2022-03-16-1432-pacific-distro-default-smithi/6742149

I did a ``dnf info runc`` on smithi161 and got:

Installed Packages
Name         : runc
Version      : 1.0.0
Release      : 74.rc95.module+el8.4.0+11822+6cc1e7d7
Architecture : x86_64
Size         : 12 M
Source       : runc-1.0.0-74.rc95.module+el8.4.0+11822+6cc1e7d7.src.rpm
Repository   : @System
From repo    : rhel-8-for-x86_64-appstream-rpms
Summary      : CLI for running Open Containers
URL          : https://github.com/opencontainers/runc
License      : ASL 2.0
Description  : The runc command can be used to start containers which are packaged
             : in accordance with the Open Container Initiative's specifications,
             : and to manage containers running under runc.

This means that smithi161 contains a bad version of runc.
However, will need to confirm whether we are upgrading this package when running the test on teuthology.

Great, thanks for the feedback. From the logs RHEL 8.4 (the failing system) has a runc 1.0.0 which has the runc BUG. So as summary we need to upgrade these systems to a new version where a runc >= 1.0.1 is intalled:

• For Ubuntu 20.04 in this case it seems that it's using an old version of runc (version without the fix: 1.0.0). In this case we need to update this package on those hosts. There's an update which upgrades this package to 1.0.1 version (runc 1.0.1-0ubuntu2~20.04.1).

• For RHEL based systems: we need to upgrade to RHEL 8.5 which has a fixed version of runc.

As a side note, it seems that container-tools:3.0 don't solve the issue (runc <= 1.0.0)

runc-0:1.0.0-70.rc92.module_el8.5.0+736+58cc1a5a.x86_64
Name             : container-tools
Stream           : 3.0

It seems that the issue has been solved in 4.0 version:

runc-1:1.0.3-1.module_el8.6.0+1108+b13568aa.x86_64
Name             : container-tools
Stream           : 4.0

find . -name \*.gz -print0 | xargs -0 zgrep runc

./remote/smithi161/syslog/misc.log.gz:2022-03-17T15:28:45.533580+00:00 smithi161 systemd[1220]: run-runc-2598a76ebf4629e218443941242f84e89b73d9877d995a5ef8a354ccbd382f25-runc.ezzlAK.mount: Succeeded.
./remote/smithi161/syslog/misc.log.gz:2022-03-17T15:29:06.948465+00:00 smithi161 systemd[1220]: run-runc-7d7e74794b404956bf07b6da9dc33fb4289fc8b360edbdc74aa65f62c1e37146-runc.qJfPMO.mount: Succeeded.
./remote/smithi161/syslog/misc.log.gz:2022-03-17T15:29:12.881841+00:00 smithi161 systemd[1220]: run-runc-91eda72b44c7fee03a02b51d8f176dc3ec9d5f7cd9d96b688c667b8c22a97f3e-runc.OMNMIj.mount: Succeeded.
./remote/smithi161/syslog/misc.log.gz:2022-03-17T15:29:20.232547+00:00 smithi161 systemd[1220]: run-runc-868f17b77e1db87bc2f4d0ac49bfa95de95ca688a0682773957819e2bfe493c9-runc.D4t6oD.mount: Succeeded.
./remote/smithi161/syslog/journalctl-b0.gz:Mar 17 15:28:45 smithi161 systemd[1220]: run-runc-2598a76ebf4629e218443941242f84e89b73d9877d995a5ef8a354ccbd382f25-runc.ezzlAK.mount: Succeeded.
./remote/smithi161/syslog/journalctl-b0.gz:Mar 17 15:29:06 smithi161 systemd[1220]: run-runc-7d7e74794b404956bf07b6da9dc33fb4289fc8b360edbdc74aa65f62c1e37146-runc.qJfPMO.mount: Succeeded.
./remote/smithi161/syslog/journalctl-b0.gz:Mar 17 15:29:12 smithi161 systemd[1220]: run-runc-91eda72b44c7fee03a02b51d8f176dc3ec9d5f7cd9d96b688c667b8c22a97f3e-runc.OMNMIj.mount: Succeeded.
./remote/smithi161/syslog/journalctl-b0.gz:Mar 17 15:29:20 smithi161 systemd[1220]: run-runc-868f17b77e1db87bc2f4d0ac49bfa95de95ca688a0682773957819e2bfe493c9-runc.D4t6oD.mount: Succeeded.
./remote/smithi161/audit/audit.log.gz:type=SOFTWARE_UPDATE msg=audit(1647530101.316:6535): pid=24695 uid=0 auid=1000 ses=6 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=update sw="runc-1.0.0-72.rc92.module+el8.4.0+12507+7f8c8fd7.x86_64" sw_type=rpm key_enforce=0 gpg_res=1 root_dir="/" comm="dnf" exe="/usr/libexec/platform-python3.6" hostname=? addr=? terminal=? res=success'UID="root" AUID="ubuntu" 
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:42.028865+00:00 smithi149 systemd[35797]: run-runc-4590c6a8d7c87a03a6da7d46109538020afb4fdf65b71625900f1954866bbb5b-runc.n6YbDF.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:43.814003+00:00 smithi149 systemd[5899]: run-runc-d4a96484e737756e29dd0dbec8b6587859990d5cf7fd8852eeb4b4e376a5dd7f-runc.x1sGSa.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:46.926538+00:00 smithi149 systemd[35797]: run-runc-c18b13cb8357be7de8e646883bf78b16f943f8de8299bd80ba332f87f0ef8ff9-runc.acaU4D.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:48.143213+00:00 smithi149 systemd[5899]: run-runc-bede0dd006c78b2406ccee63bf7220e63065b1107fad52cebfceec488e0c2b58-runc.wsXERp.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:55.659494+00:00 smithi149 systemd[35797]: run-runc-21743e3191ce7971f9fb1e531b28271dee09a6f419e1eb2323c7318148a5ece3-runc.71Aeky.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:58.517742+00:00 smithi149 systemd[5899]: run-runc-4c6514fb173e06fbec0e7935585f823827e067a904ffdac5966112326558facb-runc.VeH6Om.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:59.967485+00:00 smithi149 systemd[35797]: run-runc-18b49bcd0416e6354d4d0da48a9a84ae26dc37c11101c62cbf41ee73c76ffe62-runc.W0G6Ig.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:28:03.912917+00:00 smithi149 systemd[35797]: run-runc-00506935793f97802fc22dbf876e9e912183c51408289947dc8b4f5df426b4f3-runc.EJ6X2N.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:28:09.027983+00:00 smithi149 systemd[5899]: run-runc-304f139134730de92147c5a8bb962bdc77168aa3a32d1734c5e17cd4df9f4dd4-runc.KC89pR.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:29:02.771368+00:00 smithi149 systemd[35797]: run-runc-dc7e12909f8ca080c7acadee4b56920ec561e39deccb72134c6b6c6c076be907-runc.lqxg0R.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:29:03.794449+00:00 smithi149 systemd[35797]: run-runc-deaa644728d4e402b6833292219a5c41d6cb751d8623881187e3b890d887cd9e-runc.73M37n.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:42 smithi149 systemd[35797]: run-runc-4590c6a8d7c87a03a6da7d46109538020afb4fdf65b71625900f1954866bbb5b-runc.n6YbDF.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:43 smithi149 systemd[5899]: run-runc-d4a96484e737756e29dd0dbec8b6587859990d5cf7fd8852eeb4b4e376a5dd7f-runc.x1sGSa.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:46 smithi149 systemd[35797]: run-runc-c18b13cb8357be7de8e646883bf78b16f943f8de8299bd80ba332f87f0ef8ff9-runc.acaU4D.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:48 smithi149 systemd[5899]: run-runc-bede0dd006c78b2406ccee63bf7220e63065b1107fad52cebfceec488e0c2b58-runc.wsXERp.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:55 smithi149 systemd[35797]: run-runc-21743e3191ce7971f9fb1e531b28271dee09a6f419e1eb2323c7318148a5ece3-runc.71Aeky.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:58 smithi149 systemd[5899]: run-runc-4c6514fb173e06fbec0e7935585f823827e067a904ffdac5966112326558facb-runc.VeH6Om.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:59 smithi149 systemd[35797]: run-runc-18b49bcd0416e6354d4d0da48a9a84ae26dc37c11101c62cbf41ee73c76ffe62-runc.W0G6Ig.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:28:03 smithi149 systemd[35797]: run-runc-00506935793f97802fc22dbf876e9e912183c51408289947dc8b4f5df426b4f3-runc.EJ6X2N.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:28:09 smithi149 systemd[5899]: run-runc-304f139134730de92147c5a8bb962bdc77168aa3a32d1734c5e17cd4df9f4dd4-runc.KC89pR.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:29:02 smithi149 systemd[35797]: run-runc-dc7e12909f8ca080c7acadee4b56920ec561e39deccb72134c6b6c6c076be907-runc.lqxg0R.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:29:03 smithi149 systemd[35797]: run-runc-deaa644728d4e402b6833292219a5c41d6cb751d8623881187e3b890d887cd9e-runc.73M37n.mount: Succeeded.
./remote/smithi149/audit/audit.log.gz:type=SOFTWARE_UPDATE msg=audit(1647530102.692:6531): pid=24691 uid=0 auid=1000 ses=6 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=update sw="runc-1.0.0-72.rc92.module+el8.4.0+12507+7f8c8fd7.x86_64" sw_type=rpm key_enforce=0 gpg_res=1 root_dir="/" comm="dnf" exe="/usr/libexec/platform-python3.6" hostname=? addr=? terminal=? res=success'UID="root" AUID="ubuntu" 

The audit.log.gz contains a SOFTWARE_UPDATE sw="runc-1.0.0-72.rc92.module+el8.4.0+12507+7f8c8fd7.x86_64".
Not sure if it means we are updating the runc or is just using the installed version.

/a/nojha-2022-03-25_19:43:39-rados-wip-quincy-fast-shutdown-backports-distro-basic-smithi/6762077

/a/yuriw-2022-05-09_15:52:46-rados-wip-yuri6-testing-2022-05-09-0734-octopus-distro-default-smithi/6828876

/a/yuriw-2022-06-01_02:28:14-rados-wip-yuri-testing-2022-05-31-1642-quincy-distro-default-smithi/6856868

https://pulpito.ceph.com/adking-2022-06-23_22:33:27-orch:cephadm-wip-adk3-testing-2022-06-23-1416-quincy-distro-default-smithi/6895721/
https://pulpito.ceph.com/adking-2022-06-23_22:33:27-orch:cephadm-wip-adk3-testing-2022-06-23-1416-quincy-distro-default-smithi/6895683

/a/yuriw-2022-07-19_23:25:12-rados-wip-yuri2-testing-2022-07-15-0755-pacific-distro-default-smithi/6939035

/a/yuriw-2022-08-22_21:19:34-rados-wip-yuri4-testing-2022-08-18-1020-pacific-distro-default-smithi/6986483

/a/yuriw-2022-08-22_21:19:34-rados-wip-yuri4-testing-2022-08-18-1020-pacific-distro-default-smithi/6986484

2022-08-22T23:21:37.867 INFO:teuthology.orchestra.run.smithi099.stderr:Error: OCI runtime error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: Unit libpod-390f634f209fb08e43b42367985b127084dfa3f0c599bcacda440b7e8b6576d1.scope not found.
2022-08-22T23:21:37.889 DEBUG:teuthology.orchestra.run:got remote process result: 127

/a/yuriw-2022-08-17_19:34:54-rados-wip-yuri7-testing-2022-08-17-0943-quincy-distro-default-smithi/6977626

/a/yuriw-2022-08-24_14:12:57-rados-wip-yuri-testing-2022-08-23-1120-quincy-distro-default-smithi/6989191

I've already provided a detailed explanation of the root cause of this issue. Please consider upgrading the needed sw to fix the test case.

yuriw-2022-09-27_23:37:28-rados-wip-yuri2-testing-2022-09-27-1455-distro-default-smithi/7046226

/a/yuriw-2022-12-22_15:18:04-rados-pacific_16.2.11_RC3-distro-default-smithi/7124805

/a/ksirivad-2022-12-22_17:58:01-rados-wip-ksirivad-testing-pacific-distro-default-smithi/7125089/

/a/yuriw-2023-01-27_01:09:38-rados-wip-yuri2-testing-2023-01-26-1532-distro-default-smithi/7140569

/a/yuriw-2023-02-03_23:37:14-rados-wip-yuri4-testing-2023-02-03-1341-pacific-distro-default-smithi/7148433

/a/lflores-2023-03-27_02:17:31-rados-wip-aclamk-bs-elastic-shared-blob-save-25.03.2023-a-distro-default-smithi/7220942

/a/yuriw-2023-04-04_15:24:40-rados-wip-yuri4-testing-2023-03-31-1237-distro-default-smithi/7231172

/a/yuriw-2023-04-04_21:18:37-rados-wip-yuri3-testing-2023-04-04-0833-pacific-distro-default-smithi/7231900

/a/yuriw-2023-04-25_14:15:40-rados-pacific-release-distro-default-smithi/7251234

/a/yuriw-2023-05-06_14:41:44-rados-pacific-release-distro-default-smithi/7264492

/a/lflores-2023-05-03_18:16:04-rados-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7261527

/a/yuriw-2023-04-26_01:16:19-rados-wip-yuri11-testing-2023-04-25-1605-pacific-distro-default-smithi/7254092

/a/yuriw-2023-05-10_18:54:50-rados-wip-yuri-testing-2023-05-10-0917-distro-default-smithi/7270551

/a/yuriw-2023-05-10_14:47:51-rados-wip-yuri5-testing-2023-05-09-1324-pacific-distro-default-smithi/7269686

/a/yuriw-2023-05-19_15:13:10-rados-wip-yuri10-testing-2023-05-18-0815-quincy-distro-default-smithi/7278647

/a/yuriw-2023-05-22_23:22:00-rados-wip-yuri-testing-2023-05-22-0845-reef-distro-default-smithi/7282870

/a/yuriw-2023-05-30_20:25:48-rados-wip-yuri5-testing-2023-05-30-0828-quincy-distro-default-smithi/7290521

/a/yuriw-2023-07-19_14:33:14-rados-wip-yuri11-testing-2023-07-18-0927-pacific-distro-default-smithi/7343732

/a/yuriw-2023-07-27_22:37:12-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7354837
/a/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353554
/a/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353703

/a/yuriw-2023-07-25_21:35:32-rados-wip-yuri8-testing-2023-07-24-0819-quincy-distro-default-smithi/7352399

/a/yuriw-2023-08-16_22:40:18-rados-wip-yuri2-testing-2023-08-16-1142-pacific-distro-default-smithi/7370822

/a/yuriw-2023-08-21_23:10:07-rados-pacific-release-distro-default-smithi/7375363