Project

General

Profile

Bug #49287

podman: 'OCI runtime error' or 'Unable to find group disk'

Added by Sebastian Wagner 3 months ago. Updated about 1 month ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
cephadm
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-02-12T16:27:55.195 INFO:teuthology.orchestra.run.smithi014.stderr:Non-zero exit code 127 from /bin/podman run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:52fc503cf18cf3bb446b840ba00be073017b8373 -e NODE_NAME=smithi014 quay.ceph.io/ceph-ci/ceph:52fc503cf18cf3bb446b840ba0
0be073017b8373 -c %u %g /var/lib/ceph
2021-02-12T16:27:55.195 INFO:teuthology.orchestra.run.smithi014.stderr:stat: stderr Error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: Unit libpod-056038e1126191fba41d8a037275136f2d7aeec9710b9ee
ff792c06d8544b983.scope not found.: OCI runtime error
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:Traceback (most recent call last):
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 7697, in <module>
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:    main()
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 7686, in main
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:    r = ctx.func(ctx)
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1566, in _infer_fsid
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:    return func(ctx)
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1603, in _infer_config
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:    return func(ctx)
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1650, in _infer_image
2021-02-12T16:27:55.201 INFO:teuthology.orchestra.run.smithi014.stderr:    return func(ctx)
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 4128, in command_shell
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:    make_log_dir(ctx, ctx.fsid)
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1752, in make_log_dir
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:    uid, gid = extract_uid_gid(ctx)
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:  File "/home/ubuntu/cephtest/cephadm", line 2428, in extract_uid_gid
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:    raise RuntimeError('uid/gid not found')
2021-02-12T16:27:55.202 INFO:teuthology.orchestra.run.smithi014.stderr:RuntimeError: uid/gid not found

https://pulpito.ceph.com/swagner-2021-02-11_11:00:52-rados:cephadm-wip-swagner3-testing-2021-02-10-1322-distro-basic-smithi/5874630

History

#1 Updated by Sage Weil 3 months ago

2021-02-25T18:53:57.786 INFO:teuthology.orchestra.run.smithi141.stderr:Non-zero exit code 127 from /bin/podman run --rm --ipc=host --net=host --entrypoint stat -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:cf3694ad8a53ad63a49b370c47fe0396f994a744 -e NODE_NAME=smithi141 quay.ceph.io/ceph-ci/ceph:cf3694ad8a53ad63a49b370c47fe0396f994a744 -c %u %g /var/lib/ceph
2021-02-25T18:53:57.795 INFO:teuthology.orchestra.run.smithi141.stderr:stat: stderr Error: OCI runtime error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: Unit libpod-224e2eaa6207f5d6942dea5588c93063988b2ea02d99398af2f439ce823cf1a0.scope not found.

/a/sage-2021-02-25_18:30:47-rados:cephadm-wip-sage-testing-2021-02-25-1102-distro-basic-smithi/5913497

#2 Updated by Sage Weil 2 months ago

/a/sage-2021-03-01_20:25:17-rados-wip-sage4-testing-2021-03-01-1042-distro-basic-gibba/5924180

description: rados/cephadm/with-work/{distro/ubuntu_20.04_podman_testing fixed-2 mode/packaged
mon_election/classic msgr/async-v1only start tasks/rados_python}

#3 Updated by Sage Weil 2 months ago

  • Subject changed from container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: Unit libpod-....scope not found.: OCI runtime error to podman 3.0.1 on 20.04: 'OCI runtime error' or 'Unable to find group disk'

when running tests on gibba, it's always 20.04, and podman 3.0.1-2. and it seems pretty consistent.

error is sometimes different though:

2021-03-02T06:04:58.911 INFO:teuthology.orchestra.run.gibba036.stderr:Error: error looking up supplemental groups for container 97e0255487e7509b635629e7bd57231c66cc287948db922d71ed380ec81101c0: Unable to find group disk

/a/sage-2021-03-01_20:25:17-rados-wip-sage4-testing-2021-03-01-1042-distro-basic-gibba/5924892
/a/sage-2021-03-01_20:25:17-rados-wip-sage4-testing-2021-03-01-1042-distro-basic-gibba/5924753

#4 Updated by Sage Weil 2 months ago

  • Subject changed from podman 3.0.1 on 20.04: 'OCI runtime error' or 'Unable to find group disk' to podman: 'OCI runtime error' or 'Unable to find group disk'

#5 Updated by Deepika Upadhyay about 1 month ago

/ceph/teuthology-archive/yuriw-2021-03-24_23:05:33-rados-wip-yuri2-testing-2021-03-24-1212-octopus-distro-basic-smithi/5995730/teuthology.log

2021-03-27T10:19:01.795 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:bc61c498949fff8c3da535d41cd130d28d16cdc3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7d65ef44-8ee5-11eb-ab2c-001a4aab830c -- ceph orch daemon add mon smithi148:172.21.15.148=smithi148
2021-03-27T10:19:04.634 INFO:teuthology.orchestra.run.smithi148.stderr:Error: OCI runtime error: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: Unit libpod-6400e71e1d28e76e078f9e2a63b55433e3c4bc8d5df6b6bb0e2076c30605eb6b.scope not found.
2021-03-27T10:19:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 127

Also available in: Atom PDF