Bug #44587: failed to write <pid> to cgroup.procs: - Orchestrator - Ceph

Custom queries

Bug queue
Bug triage
Crash queue
Crash triage
Feedback
My issues
Need Review
Pending backports
Product Backlog Scrub

Actions

Copy link

Bug #44587

open

failed to write <pid> to cgroup.procs:

Added by Sage Weil about 4 years ago. Updated 8 months ago.

Status:

New

Priority:

Normal

Assignee:

Category:

cephadm (binary)

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2020-03-13T01:55:00.607 INFO:ceph.prometheus.a.smithi166.stdout:Mar 13 01:55:00 smithi166 systemd[1]: Starting Ceph prometheus.a for eee3c39c-64cc-11ea-9a47-001a4aab830c...
2020-03-13T01:55:00.725 INFO:teuthology.orchestra.run.smithi166.stderr:Error: container_linux.go:346: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"failed to write 34473 to cgroup.procs: open /sys/fs/cgroup/blkio/machine.slice/libpod-bfcc3e5be38bb3da79931ab84b072b089d34acc75a8b70bc882f473680ebf672.scope/cgroup.procs: no such file or directory\"": OCI runtime command not found error

/a/sage-2020-03-13_01:29:38-rados:cephadm-wip-sage3-testing-2020-03-12-1708-distro-basic-smithi/4851460

Related issues 1 (0 open — 1 closed)

Has duplicate Ceph - Bug #50280: cephadm: RuntimeError: uid/gid not found

Duplicate

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by Sebastian Wagner almost 4 years ago

Source set to Q/A

Actions

Copy link

Updated by Sebastian Wagner over 3 years ago

Status changed from New to Can't reproduce

Actions

Copy link

Updated by Sebastian Wagner about 3 years ago

Has duplicate Bug #50280: cephadm: RuntimeError: uid/gid not found added

Actions

Copy link

Updated by Sebastian Wagner about 3 years ago

Status changed from Can't reproduce to New

Actions

Copy link

Updated by Sebastian Wagner almost 3 years ago

Category set to cephadm (binary)

Actions

Copy link

Updated by Sebastian Wagner over 2 years ago

https://pulpito.ceph.com/swagner-2021-09-07_13:27:36-rados:cephadm-wip-swagner-testing-2021-09-07-1243-distro-basic-smithi/6377995/

Actions

Copy link

Updated by Adam King over 2 years ago

When testing agent

https://pulpito.ceph.com/adking-2021-09-13_22:25:39-rados:cephadm-wip-adk-testing-2021-09-13-1408-distro-basic-smithi/6388540/
https://pulpito.ceph.com/adking-2021-09-13_22:25:39-rados:cephadm-wip-adk-testing-2021-09-13-1408-distro-basic-smithi/6388548/

Actions

Copy link

Updated by Sebastian Wagner over 2 years ago

caused by a downgrade of podman in sepia.

Actions

Copy link

Updated by Neha Ojha over 2 years ago

2021-12-07T19:40:29.616 INFO:teuthology.orchestra.run.smithi174.stderr:Non-zero exit code 126 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:02e20d0396c1e49054be17c5f0527ae529a28c91 -e NODE_NAME=smithi174 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:02e20d0396c1e49054be17c5f0527ae529a28c91 -c %u %g /var/lib/ceph
2021-12-07T19:40:29.617 INFO:teuthology.orchestra.run.smithi174.stderr:stat: stderr Error: OCI runtime error: container_linux.go:370: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: failed to write 33700 to cgroup.procs: write /sys/fs/cgroup/devices/machine.slice/libpod-dc31f7868d43ce593f65a3f010c9ddf13a2ce45c47eab28e2a6f0d99f5a009b9.scope/cgroup.procs: no such device
2021-12-07T19:40:29.622 INFO:teuthology.orchestra.run.smithi174.stderr:Traceback (most recent call last):
2021-12-07T19:40:29.623 INFO:teuthology.orchestra.run.smithi174.stderr:  File "/home/ubuntu/cephtest/cephadm", line 8339, in <module>
2021-12-07T19:40:29.623 INFO:teuthology.orchestra.run.smithi174.stderr:    main()
2021-12-07T19:40:29.623 INFO:teuthology.orchestra.run.smithi174.stderr:  File "/home/ubuntu/cephtest/cephadm", line 8327, in main
2021-12-07T19:40:29.623 INFO:teuthology.orchestra.run.smithi174.stderr:    r = ctx.func(ctx)
2021-12-07T19:40:29.623 INFO:teuthology.orchestra.run.smithi174.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1719, in _infer_config
2021-12-07T19:40:29.624 INFO:teuthology.orchestra.run.smithi174.stderr:    return func(ctx)
2021-12-07T19:40:29.624 INFO:teuthology.orchestra.run.smithi174.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1676, in _infer_fsid
2021-12-07T19:40:29.624 INFO:teuthology.orchestra.run.smithi174.stderr:    return func(ctx)
2021-12-07T19:40:29.624 INFO:teuthology.orchestra.run.smithi174.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1719, in _infer_config
2021-12-07T19:40:29.625 INFO:teuthology.orchestra.run.smithi174.stderr:    return func(ctx)
2021-12-07T19:40:29.625 INFO:teuthology.orchestra.run.smithi174.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1763, in _infer_image
2021-12-07T19:40:29.625 INFO:teuthology.orchestra.run.smithi174.stderr:    return func(ctx)
2021-12-07T19:40:29.625 INFO:teuthology.orchestra.run.smithi174.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1663, in _validate_fsid
2021-12-07T19:40:29.626 INFO:teuthology.orchestra.run.smithi174.stderr:    return func(ctx)
2021-12-07T19:40:29.626 INFO:teuthology.orchestra.run.smithi174.stderr:  File "/home/ubuntu/cephtest/cephadm", line 5142, in command_shell
2021-12-07T19:40:29.626 INFO:teuthology.orchestra.run.smithi174.stderr:    make_log_dir(ctx, ctx.fsid)
2021-12-07T19:40:29.626 INFO:teuthology.orchestra.run.smithi174.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1865, in make_log_dir
2021-12-07T19:40:29.626 INFO:teuthology.orchestra.run.smithi174.stderr:    uid, gid = extract_uid_gid(ctx)
2021-12-07T19:40:29.627 INFO:teuthology.orchestra.run.smithi174.stderr:  File "/home/ubuntu/cephtest/cephadm", line 2619, in extract_uid_gid
2021-12-07T19:40:29.627 INFO:teuthology.orchestra.run.smithi174.stderr:    raise RuntimeError('uid/gid not found')
2021-12-07T19:40:29.627 INFO:teuthology.orchestra.run.smithi174.stderr:RuntimeError: uid/gid not found

/a/yuriw-2021-12-07_16:04:59-rados-wip-yuri5-testing-2021-12-06-1619-distro-default-smithi/6550593

Actions

Copy link

#10

Updated by Aishwarya Mathuria over 2 years ago

/a/yuriw-2022-01-13_18:06:52-rados-wip-yuri3-testing-2022-01-13-0809-distro-default-smithi/6614461

Actions

Copy link

#11

Updated by Redouane Kachach Elhichou about 2 years ago

Priority changed from High to Normal

Actions

Copy link

#12

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

Subject changed from failed to write 34473 to cgroup.procs: to failed to write <pid> to cgroup.procs:

/a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6944442/

2022-07-23T00:44:33.389 INFO:journalctl@ceph.mon.b.smithi170.stdout:Jul 23 00:44:33 smithi170 systemd[1]: Starting Ceph mon.b for 3911b92e-0a20-11ed-842f-001a4aab830c...
2022-07-23T00:44:33.684 INFO:teuthology.orchestra.run.smithi170.stderr:Error: OCI runtime error: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: failed to write 90338 to cgroup.procs: write /sys/fs/cgroup/memory/machine.slice/libpod-a48e01ceb59134e5039749a70089fc9bd8cbded60f9a5ee9c7a2dcd45778ab66.scope/cgroup.procs: no such device
2022-07-23T00:44:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 126
2022-07-23T00:44:33.713 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/contextutil.py", line 31, in nested
    vars.append(enter())
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_19acc2f7e3edb197f028ffa801e28f62f3698c79/qa/tasks/cephadm.py", line 678, in ceph_mons
    stdout=StringIO(),
  File "/home/teuthworker/src/github.com_ceph_ceph-c_19acc2f7e3edb197f028ffa801e28f62f3698c79/qa/tasks/cephadm.py", line 49, in _shell
    **kwargs
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/orchestra/remote.py", line 510, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 126: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:19acc2f7e3edb197f028ffa801e28f62f3698c79 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3911b92e-0a20-11ed-842f-001a4aab830c -- ceph mon dump -f json'
2022-07-23T00:44:33.713 INFO:tasks.cephadm:Cleaning up testdir ceph.* files...

Actions

Copy link

#13