Project

General

Profile

Actions

Bug #49287

open

podman: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not found

Added by Sebastian Wagner about 3 years ago. Updated about 2 months ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
cephadm
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-02-12T16:27:55.195 INFO:teuthology.orchestra.run.smithi014.stderr:Non-zero exit code 127 from /bin/podman run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:52fc503cf18cf3bb446b840ba00be073017b8373 -e NODE_NAME=smithi014 quay.ceph.io/ceph-ci/ceph:52fc503cf18cf3bb446b840ba0
0be073017b8373 -c %u %g /var/lib/ceph
2021-02-12T16:27:55.195 INFO:teuthology.orchestra.run.smithi014.stderr:stat: stderr Error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: Unit libpod-056038e1126191fba41d8a037275136f2d7aeec9710b9ee
ff792c06d8544b983.scope not found.: OCI runtime error
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:Traceback (most recent call last):
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 7697, in <module>
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:    main()
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 7686, in main
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:    r = ctx.func(ctx)
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1566, in _infer_fsid
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:    return func(ctx)
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1603, in _infer_config
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:    return func(ctx)
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1650, in _infer_image
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:    return func(ctx)
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 4128, in command_shell
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:    make_log_dir(ctx, ctx.fsid)
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1752, in make_log_dir
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:    uid, gid = extract_uid_gid(ctx)
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 2428, in extract_uid_gid
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:    raise RuntimeError('uid/gid not found')
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:RuntimeError: uid/gid not found

https://pulpito.ceph.com/swagner-2021-02-11_11:00:52-rados:cephadm-wip-swagner3-testing-2021-02-10-1322-distro-basic-smithi/5874630


Related issues 7 (2 open5 closed)

Related to sepia - Bug #53827: cephadm exited with error code when creating osd: Input/Output error. Faulty NVME?New

Actions
Related to CephFS - Bug #65261: qa/cephfs: cephadm related failure on fs/upgrade jobNewAdam King

Actions
Has duplicate Orchestrator - Bug #51102: RuntimeError: uid/gid not found in "rados/thrash-old-clients" testDuplicate

Actions
Has duplicate Orchestrator - Bug #53681: Failed to extract uid/gid for path /var/lib/cephDuplicate

Actions
Has duplicate Orchestrator - Bug #53766: ceph orch ls: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not foundDuplicate

Actions
Has duplicate Orchestrator - Bug #61393: qa: cephadm command failedDuplicate

Actions
Has duplicate Infrastructure - Bug #58907: OCI runtime error: runc: runc create failed: unable to start container processDuplicate

Actions
Actions #1

Updated by Sage Weil about 3 years ago

2021-02-25T18:53:57.786 INFO:teuthology.orchestra.run.smithi141.stderr:Non-zero exit code 127 from /bin/podman run --rm --ipc=host --net=host --entrypoint stat -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:cf3694ad8a53ad63a49b370c47fe0396f994a744 -e NODE_NAME=smithi141 quay.ceph.io/ceph-ci/ceph:cf3694ad8a53ad63a49b370c47fe0396f994a744 -c %u %g /var/lib/ceph
2021-02-25T18:53:57.795 INFO:teuthology.orchestra.run.smithi141.stderr:stat: stderr Error: OCI runtime error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: Unit libpod-224e2eaa6207f5d6942dea5588c93063988b2ea02d99398af2f439ce823cf1a0.scope not found.

/a/sage-2021-02-25_18:30:47-rados:cephadm-wip-sage-testing-2021-02-25-1102-distro-basic-smithi/5913497
Actions #2

Updated by Sage Weil about 3 years ago

/a/sage-2021-03-01_20:25:17-rados-wip-sage4-testing-2021-03-01-1042-distro-basic-gibba/5924180

description: rados/cephadm/with-work/{distro/ubuntu_20.04_podman_testing fixed-2 mode/packaged
mon_election/classic msgr/async-v1only start tasks/rados_python}

Actions #3

Updated by Sage Weil about 3 years ago

  • Subject changed from container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: Unit libpod-....scope not found.: OCI runtime error to podman 3.0.1 on 20.04: 'OCI runtime error' or 'Unable to find group disk'

when running tests on gibba, it's always 20.04, and podman 3.0.1-2. and it seems pretty consistent.

error is sometimes different though:

2021-03-02T06:04:58.911 INFO:teuthology.orchestra.run.gibba036.stderr:Error: error looking up supplemental groups for container 97e0255487e7509b635629e7bd57231c66cc287948db922d71ed380ec81101c0: Unable to find group disk

/a/sage-2021-03-01_20:25:17-rados-wip-sage4-testing-2021-03-01-1042-distro-basic-gibba/5924892
/a/sage-2021-03-01_20:25:17-rados-wip-sage4-testing-2021-03-01-1042-distro-basic-gibba/5924753

Actions #4

Updated by Sage Weil about 3 years ago

  • Subject changed from podman 3.0.1 on 20.04: 'OCI runtime error' or 'Unable to find group disk' to podman: 'OCI runtime error' or 'Unable to find group disk'
Actions #5

Updated by Deepika Upadhyay about 3 years ago

/ceph/teuthology-archive/yuriw-2021-03-24_23:05:33-rados-wip-yuri2-testing-2021-03-24-1212-octopus-distro-basic-smithi/5995730/teuthology.log

2021-03-27T10:19:01.795 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:bc61c498949fff8c3da535d41cd130d28d16cdc3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d65ef44-8ee5-11eb-ab2c-001a4aab830c -- ceph orch daemon add mon smithi148:172.21.15.148=smithi148
2021-03-27T10:19:04.634 INFO:teuthology.orchestra.run.smithi148.stderr:Error: OCI runtime error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: Unit libpod-6400e71e1d28e76e078f9e2a63b55433e3c4bc8d5df6b6bb0e2076c30605eb6b.scope not found.
2021-03-27T10:19:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 127

Actions #6

Updated by Sebastian Wagner almost 3 years ago

  • Subject changed from podman: 'OCI runtime error' or 'Unable to find group disk' to podman: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not found
Actions #7

Updated by Sebastian Wagner almost 3 years ago

  • Has duplicate Bug #51102: RuntimeError: uid/gid not found in "rados/thrash-old-clients" test added
Actions #8

Updated by Sebastian Wagner almost 3 years ago

https://pulpito.ceph.com/pdonnell-2021-07-12_18:17:18-fs:workload-wip-pdonnell-testing-20210710.030422-distro-basic-smithi/6266629/

fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 
Actions #11

Updated by Sebastian Wagner over 2 years ago

  • Status changed from New to Resolved
Actions #12

Updated by Sebastian Wagner over 2 years ago

  • Status changed from Resolved to New

By switchin over to container-tools:3.0, we're not affected by this podman issue again.

Actions #14

Updated by Sebastian Wagner over 2 years ago

Deepika, I don't think we can do anything about this at this time.

Actions #15

Updated by Neha Ojha over 2 years ago

/a/yuriw-2021-09-16_18:23:18-rados-wip-yuri2-testing-2021-09-16-0923-distro-basic-smithi/6393507

Actions #18

Updated by Sridhar Seshasayee over 2 years ago

/a/yuriw-2021-12-03_15:27:18-rados-wip-yuri11-testing-2021-12-02-1451-distro-default-smithi/6542708

Actions #19

Updated by Sebastian Wagner over 2 years ago

/a/yuriw-2021-12-09_00:18:57-rados-wip-yuri-testing-2021-12-08-1336-distro-default-smithi/6553774/

Actions #20

Updated by Sebastian Wagner over 2 years ago

  • Has duplicate Bug #53681: Failed to extract uid/gid for path /var/lib/ceph added
Actions #21

Updated by Sebastian Wagner over 2 years ago

  • Has duplicate Bug #53766: ceph orch ls: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not found added
Actions #22

Updated by Laura Flores over 2 years ago

/a/lflores-2022-01-05_19:04:35-rados-wip-lflores-mgr-rocksdb-distro-default-smithi/6596855

Actions #23

Updated by Sridhar Seshasayee over 2 years ago

/a/yuriw-2022-01-08_17:57:43-rados-wip-yuri8-testing-2022-01-07-1541-distro-default-smithi/6603250

Actions #24

Updated by Laura Flores about 2 years ago

/a/yuriw-2022-02-17_22:49:55-rados-wip-yuri3-testing-2022-02-17-1256-distro-default-smithi/6692382

Actions #25

Updated by Kamoltat (Junior) Sirivadhna about 2 years ago

/a/yuriw-2022-02-17_23:23:56-rados-wip-yuri7-testing-2022-02-17-0852-pacific-distro-default-smithi/6692693

Actions #26

Updated by Kamoltat (Junior) Sirivadhna about 2 years ago

/a/yuriw-2022-03-17_14:54:32-rados-wip-yuri10-testing-2022-03-16-1432-pacific-distro-default-smithi/6742149

Actions #27

Updated by Redouane Kachach Elhichou about 2 years ago

Reading the BZ thread it seems that:
1- This issue is caused by a BUG in runc (https://github.com/opencontainers/runc/pull/2614) which should be solved in runc version >= 1.0.1
2- backport of the fix to container-tools 3.0 in RHEL 8.4 is tracked by https://bugzilla.redhat.com/show_bug.cgi?id=2000570

My doubt is: which versions of these packages are being installed on the teuthology machines where these tests are failed?

Can you please check which runtime is being used by podman by running:

podman info --debug

and

dnf info runc

Actions #28

Updated by Redouane Kachach Elhichou about 2 years ago

  • Assignee set to Redouane Kachach Elhichou
Actions #29

Updated by Kamoltat (Junior) Sirivadhna about 2 years ago

Hi Redouane,

as we have discussed offline,

smithi161 which was one of the machines used in the job that came across this bug:
/a/yuriw-2022-03-17_14:54:32-rados-wip-yuri10-testing-2022-03-16-1432-pacific-distro-default-smithi/6742149

I did a ``dnf info runc`` on smithi161 and got:

Installed Packages
Name         : runc
Version      : 1.0.0
Release      : 74.rc95.module+el8.4.0+11822+6cc1e7d7
Architecture : x86_64
Size         : 12 M
Source       : runc-1.0.0-74.rc95.module+el8.4.0+11822+6cc1e7d7.src.rpm
Repository   : @System
From repo    : rhel-8-for-x86_64-appstream-rpms
Summary      : CLI for running Open Containers
URL          : https://github.com/opencontainers/runc
License      : ASL 2.0
Description  : The runc command can be used to start containers which are packaged
             : in accordance with the Open Container Initiative's specifications,
             : and to manage containers running under runc.

This means that smithi161 contains a bad version of runc.
However, will need to confirm whether we are upgrading this package when running the test on teuthology.

Actions #30

Updated by Redouane Kachach Elhichou about 2 years ago

Great, thanks for the feedback. From the logs RHEL 8.4 (the failing system) has a runc 1.0.0 which has the runc BUG. So as summary we need to upgrade these systems to a new version where a runc >= 1.0.1 is intalled:

  • For Ubuntu 20.04 in this case it seems that it's using an old version of runc (version without the fix: 1.0.0). In this case we need to update this package on those hosts. There's an update which upgrades this package to 1.0.1 version (runc 1.0.1-0ubuntu2~20.04.1).
  • For RHEL based systems: we need to upgrade to RHEL 8.5 which has a fixed version of runc.

As a side note, it seems that container-tools:3.0 don't solve the issue (runc <= 1.0.0)

runc-0:1.0.0-70.rc92.module_el8.5.0+736+58cc1a5a.x86_64
Name             : container-tools
Stream           : 3.0

It seems that the issue has been solved in 4.0 version:

runc-1:1.0.3-1.module_el8.6.0+1108+b13568aa.x86_64
Name             : container-tools
Stream           : 4.0
Actions #31

Updated by Kamoltat (Junior) Sirivadhna about 2 years ago

find . -name \*.gz -print0 | xargs -0 zgrep runc

./remote/smithi161/syslog/misc.log.gz:2022-03-17T15:28:45.533580+00:00 smithi161 systemd[1220]: run-runc-2598a76ebf4629e218443941242f84e89b73d9877d995a5ef8a354ccbd382f25-runc.ezzlAK.mount: Succeeded.
./remote/smithi161/syslog/misc.log.gz:2022-03-17T15:29:06.948465+00:00 smithi161 systemd[1220]: run-runc-7d7e74794b404956bf07b6da9dc33fb4289fc8b360edbdc74aa65f62c1e37146-runc.qJfPMO.mount: Succeeded.
./remote/smithi161/syslog/misc.log.gz:2022-03-17T15:29:12.881841+00:00 smithi161 systemd[1220]: run-runc-91eda72b44c7fee03a02b51d8f176dc3ec9d5f7cd9d96b688c667b8c22a97f3e-runc.OMNMIj.mount: Succeeded.
./remote/smithi161/syslog/misc.log.gz:2022-03-17T15:29:20.232547+00:00 smithi161 systemd[1220]: run-runc-868f17b77e1db87bc2f4d0ac49bfa95de95ca688a0682773957819e2bfe493c9-runc.D4t6oD.mount: Succeeded.
./remote/smithi161/syslog/journalctl-b0.gz:Mar 17 15:28:45 smithi161 systemd[1220]: run-runc-2598a76ebf4629e218443941242f84e89b73d9877d995a5ef8a354ccbd382f25-runc.ezzlAK.mount: Succeeded.
./remote/smithi161/syslog/journalctl-b0.gz:Mar 17 15:29:06 smithi161 systemd[1220]: run-runc-7d7e74794b404956bf07b6da9dc33fb4289fc8b360edbdc74aa65f62c1e37146-runc.qJfPMO.mount: Succeeded.
./remote/smithi161/syslog/journalctl-b0.gz:Mar 17 15:29:12 smithi161 systemd[1220]: run-runc-91eda72b44c7fee03a02b51d8f176dc3ec9d5f7cd9d96b688c667b8c22a97f3e-runc.OMNMIj.mount: Succeeded.
./remote/smithi161/syslog/journalctl-b0.gz:Mar 17 15:29:20 smithi161 systemd[1220]: run-runc-868f17b77e1db87bc2f4d0ac49bfa95de95ca688a0682773957819e2bfe493c9-runc.D4t6oD.mount: Succeeded.
./remote/smithi161/audit/audit.log.gz:type=SOFTWARE_UPDATE msg=audit(1647530101.316:6535): pid=24695 uid=0 auid=1000 ses=6 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=update sw="runc-1.0.0-72.rc92.module+el8.4.0+12507+7f8c8fd7.x86_64" sw_type=rpm key_enforce=0 gpg_res=1 root_dir="/" comm="dnf" exe="/usr/libexec/platform-python3.6" hostname=? addr=? terminal=? res=success'UID="root" AUID="ubuntu" 
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:42.028865+00:00 smithi149 systemd[35797]: run-runc-4590c6a8d7c87a03a6da7d46109538020afb4fdf65b71625900f1954866bbb5b-runc.n6YbDF.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:43.814003+00:00 smithi149 systemd[5899]: run-runc-d4a96484e737756e29dd0dbec8b6587859990d5cf7fd8852eeb4b4e376a5dd7f-runc.x1sGSa.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:46.926538+00:00 smithi149 systemd[35797]: run-runc-c18b13cb8357be7de8e646883bf78b16f943f8de8299bd80ba332f87f0ef8ff9-runc.acaU4D.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:48.143213+00:00 smithi149 systemd[5899]: run-runc-bede0dd006c78b2406ccee63bf7220e63065b1107fad52cebfceec488e0c2b58-runc.wsXERp.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:55.659494+00:00 smithi149 systemd[35797]: run-runc-21743e3191ce7971f9fb1e531b28271dee09a6f419e1eb2323c7318148a5ece3-runc.71Aeky.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:58.517742+00:00 smithi149 systemd[5899]: run-runc-4c6514fb173e06fbec0e7935585f823827e067a904ffdac5966112326558facb-runc.VeH6Om.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:27:59.967485+00:00 smithi149 systemd[35797]: run-runc-18b49bcd0416e6354d4d0da48a9a84ae26dc37c11101c62cbf41ee73c76ffe62-runc.W0G6Ig.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:28:03.912917+00:00 smithi149 systemd[35797]: run-runc-00506935793f97802fc22dbf876e9e912183c51408289947dc8b4f5df426b4f3-runc.EJ6X2N.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:28:09.027983+00:00 smithi149 systemd[5899]: run-runc-304f139134730de92147c5a8bb962bdc77168aa3a32d1734c5e17cd4df9f4dd4-runc.KC89pR.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:29:02.771368+00:00 smithi149 systemd[35797]: run-runc-dc7e12909f8ca080c7acadee4b56920ec561e39deccb72134c6b6c6c076be907-runc.lqxg0R.mount: Succeeded.
./remote/smithi149/syslog/misc.log.gz:2022-03-17T15:29:03.794449+00:00 smithi149 systemd[35797]: run-runc-deaa644728d4e402b6833292219a5c41d6cb751d8623881187e3b890d887cd9e-runc.73M37n.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:42 smithi149 systemd[35797]: run-runc-4590c6a8d7c87a03a6da7d46109538020afb4fdf65b71625900f1954866bbb5b-runc.n6YbDF.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:43 smithi149 systemd[5899]: run-runc-d4a96484e737756e29dd0dbec8b6587859990d5cf7fd8852eeb4b4e376a5dd7f-runc.x1sGSa.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:46 smithi149 systemd[35797]: run-runc-c18b13cb8357be7de8e646883bf78b16f943f8de8299bd80ba332f87f0ef8ff9-runc.acaU4D.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:48 smithi149 systemd[5899]: run-runc-bede0dd006c78b2406ccee63bf7220e63065b1107fad52cebfceec488e0c2b58-runc.wsXERp.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:55 smithi149 systemd[35797]: run-runc-21743e3191ce7971f9fb1e531b28271dee09a6f419e1eb2323c7318148a5ece3-runc.71Aeky.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:58 smithi149 systemd[5899]: run-runc-4c6514fb173e06fbec0e7935585f823827e067a904ffdac5966112326558facb-runc.VeH6Om.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:27:59 smithi149 systemd[35797]: run-runc-18b49bcd0416e6354d4d0da48a9a84ae26dc37c11101c62cbf41ee73c76ffe62-runc.W0G6Ig.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:28:03 smithi149 systemd[35797]: run-runc-00506935793f97802fc22dbf876e9e912183c51408289947dc8b4f5df426b4f3-runc.EJ6X2N.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:28:09 smithi149 systemd[5899]: run-runc-304f139134730de92147c5a8bb962bdc77168aa3a32d1734c5e17cd4df9f4dd4-runc.KC89pR.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:29:02 smithi149 systemd[35797]: run-runc-dc7e12909f8ca080c7acadee4b56920ec561e39deccb72134c6b6c6c076be907-runc.lqxg0R.mount: Succeeded.
./remote/smithi149/syslog/journalctl-b0.gz:Mar 17 15:29:03 smithi149 systemd[35797]: run-runc-deaa644728d4e402b6833292219a5c41d6cb751d8623881187e3b890d887cd9e-runc.73M37n.mount: Succeeded.
./remote/smithi149/audit/audit.log.gz:type=SOFTWARE_UPDATE msg=audit(1647530102.692:6531): pid=24691 uid=0 auid=1000 ses=6 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=update sw="runc-1.0.0-72.rc92.module+el8.4.0+12507+7f8c8fd7.x86_64" sw_type=rpm key_enforce=0 gpg_res=1 root_dir="/" comm="dnf" exe="/usr/libexec/platform-python3.6" hostname=? addr=? terminal=? res=success'UID="root" AUID="ubuntu" 

The audit.log.gz contains a SOFTWARE_UPDATE sw="runc-1.0.0-72.rc92.module+el8.4.0+12507+7f8c8fd7.x86_64".
Not sure if it means we are updating the runc or is just using the installed version.

Actions #32

Updated by Laura Flores about 2 years ago

/a/nojha-2022-03-25_19:43:39-rados-wip-quincy-fast-shutdown-backports-distro-basic-smithi/6762077

Actions #33

Updated by Laura Flores almost 2 years ago

/a/yuriw-2022-05-09_15:52:46-rados-wip-yuri6-testing-2022-05-09-0734-octopus-distro-default-smithi/6828876

Actions #34

Updated by Laura Flores almost 2 years ago

/a/yuriw-2022-06-01_02:28:14-rados-wip-yuri-testing-2022-05-31-1642-quincy-distro-default-smithi/6856868

Actions #36

Updated by Laura Flores almost 2 years ago

/a/yuriw-2022-07-19_23:25:12-rados-wip-yuri2-testing-2022-07-15-0755-pacific-distro-default-smithi/6939035

Actions #37

Updated by Matan Breizman over 1 year ago

/a/yuriw-2022-08-22_21:19:34-rados-wip-yuri4-testing-2022-08-18-1020-pacific-distro-default-smithi/6986483

Actions #38

Updated by Matan Breizman over 1 year ago

/a/yuriw-2022-08-22_21:19:34-rados-wip-yuri4-testing-2022-08-18-1020-pacific-distro-default-smithi/6986484

2022-08-22T23:21:37.867 INFO:teuthology.orchestra.run.smithi099.stderr:Error: OCI runtime error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: Unit libpod-390f634f209fb08e43b42367985b127084dfa3f0c599bcacda440b7e8b6576d1.scope not found.
2022-08-22T23:21:37.889 DEBUG:teuthology.orchestra.run:got remote process result: 127
Actions #39

Updated by Adam King over 1 year ago

  • Translation missing: en.field_tag_list set to test-failure
Actions #40

Updated by Laura Flores over 1 year ago

/a/yuriw-2022-08-17_19:34:54-rados-wip-yuri7-testing-2022-08-17-0943-quincy-distro-default-smithi/6977626

Actions #41

Updated by Laura Flores over 1 year ago

/a/yuriw-2022-08-24_14:12:57-rados-wip-yuri-testing-2022-08-23-1120-quincy-distro-default-smithi/6989191

Actions #42

Updated by Laura Flores over 1 year ago

  • Backport set to /a/yuriw-2022-09-20_17:39:55-rados-wip-yuri5-testing-2022-09-19-1007-pacific-distro-default-smithi/7039249

/a/yuriw-2022-09-20_17:39:55-rados-wip-yuri5-testing-2022-09-19-1007-pacific-distro-default-smithi/7039249

Actions #43

Updated by Redouane Kachach Elhichou over 1 year ago

I've already provided a detailed explanation of the root cause of this issue. Please consider upgrading the needed sw to fix the test case.

Actions #44

Updated by Redouane Kachach Elhichou over 1 year ago

  • Assignee deleted (Redouane Kachach Elhichou)
Actions #45

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

yuriw-2022-09-27_23:37:28-rados-wip-yuri2-testing-2022-09-27-1455-distro-default-smithi/7046226

Actions #46

Updated by Matan Breizman over 1 year ago

/a/yuriw-2022-12-22_15:18:04-rados-pacific_16.2.11_RC3-distro-default-smithi/7124805

Actions #47

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/ksirivad-2022-12-22_17:58:01-rados-wip-ksirivad-testing-pacific-distro-default-smithi/7125089/

Actions #48

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-01-27_01:09:38-rados-wip-yuri2-testing-2023-01-26-1532-distro-default-smithi/7140569

Actions #49

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-02-03_23:37:14-rados-wip-yuri4-testing-2023-02-03-1341-pacific-distro-default-smithi/7148433

Actions #50

Updated by Laura Flores about 1 year ago

/a/lflores-2023-03-27_02:17:31-rados-wip-aclamk-bs-elastic-shared-blob-save-25.03.2023-a-distro-default-smithi/7220942

Actions #51

Updated by Laura Flores about 1 year ago

  • Backport deleted (/a/yuriw-2022-09-20_17:39:55-rados-wip-yuri5-testing-2022-09-19-1007-pacific-distro-default-smithi/7039249)

/a/yuriw-2023-03-24_13:25:17-rados-quincy-release-distro-default-smithi/7219107

Actions #52

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-04-04_15:24:40-rados-wip-yuri4-testing-2023-03-31-1237-distro-default-smithi/7231172

Actions #53

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-04-04_21:18:37-rados-wip-yuri3-testing-2023-04-04-0833-pacific-distro-default-smithi/7231900

Actions #54

Updated by Laura Flores about 1 year ago

  • Related to Bug #53827: cephadm exited with error code when creating osd: Input/Output error. Faulty NVME? added
Actions #55

Updated by Laura Flores 12 months ago

/a/yuriw-2023-04-25_14:15:40-rados-pacific-release-distro-default-smithi/7251234

Actions #56

Updated by Laura Flores 12 months ago

/a/yuriw-2023-05-06_14:41:44-rados-pacific-release-distro-default-smithi/7264492

Actions #57

Updated by Laura Flores 12 months ago

/a/lflores-2023-05-03_18:16:04-rados-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7261527

Actions #58

Updated by Laura Flores 12 months ago

/a/yuriw-2023-04-26_01:16:19-rados-wip-yuri11-testing-2023-04-25-1605-pacific-distro-default-smithi/7254092

Actions #59

Updated by Laura Flores 11 months ago

/a/yuriw-2023-05-10_18:54:50-rados-wip-yuri-testing-2023-05-10-0917-distro-default-smithi/7270551

Actions #60

Updated by Laura Flores 11 months ago

/a/yuriw-2023-05-10_14:47:51-rados-wip-yuri5-testing-2023-05-09-1324-pacific-distro-default-smithi/7269686

Actions #61

Updated by Laura Flores 11 months ago

/a/yuriw-2023-05-19_15:13:10-rados-wip-yuri10-testing-2023-05-18-0815-quincy-distro-default-smithi/7278647

Actions #62

Updated by Laura Flores 11 months ago

/a/yuriw-2023-05-22_23:22:00-rados-wip-yuri-testing-2023-05-22-0845-reef-distro-default-smithi/7282870

Actions #63

Updated by Laura Flores 11 months ago

  • Has duplicate Bug #61393: qa: cephadm command failed added
Actions #64

Updated by Laura Flores 11 months ago

/a/yuriw-2023-05-30_20:25:48-rados-wip-yuri5-testing-2023-05-30-0828-quincy-distro-default-smithi/7290521

Actions #65

Updated by Laura Flores 9 months ago

/a/yuriw-2023-07-19_14:33:14-rados-wip-yuri11-testing-2023-07-18-0927-pacific-distro-default-smithi/7343732

Actions #66

Updated by Sridhar Seshasayee 9 months ago

/a/yuriw-2023-07-27_22:37:12-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7354837
/a/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353554
/a/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353703

Actions #67

Updated by Laura Flores 9 months ago

/a/yuriw-2023-07-25_21:35:32-rados-wip-yuri8-testing-2023-07-24-0819-quincy-distro-default-smithi/7352399

Actions #68

Updated by Aishwarya Mathuria 8 months ago

/a/yuriw-2023-08-16_22:40:18-rados-wip-yuri2-testing-2023-08-16-1142-pacific-distro-default-smithi/7370822

Actions #69

Updated by Laura Flores 8 months ago

/a/yuriw-2023-08-21_23:10:07-rados-pacific-release-distro-default-smithi/7375363

Actions #70

Updated by Kamoltat (Junior) Sirivadhna 3 months ago

/a/yuriw-2024-01-10_15:06:10-rados-wip-yuri-testing-2024-01-09-1331-distro-default-smithi/7511814/

Actions #71

Updated by Laura Flores 3 months ago

  • Has duplicate Bug #58907: OCI runtime error: runc: runc create failed: unable to start container process added
Actions #72

Updated by Laura Flores 3 months ago

/a/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535097

Actions #73

Updated by Laura Flores about 2 months ago

/a/yuriw-2024-03-04_20:52:58-rados-reef-release-distro-default-smithi/7581730

Actions #74

Updated by Rishabh Dave 24 days ago

  • Related to Bug #65261: qa/cephfs: cephadm related failure on fs/upgrade job added
Actions

Also available in: Atom PDF