Bug #46704
container_linux.go:349: "exec: \"stat\": executable file not found
0%
Description
Description: rados:cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start}
2020-07-24T18:11:43.836 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:43 smithi187 bash[12265]: cluster 2020-07-24T18:11:43.342367+0000 mon.smithi053 (mon.0) 263 : cluster [WRN] overall HEALTH_WARN failed to probe daemons or devices; Reduced data availability: 1 pg inactive 2020-07-24T18:11:44.617 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jul 24 18:11:44 smithi053 bash[12319]: audit 2020-07-24T18:11:44.284616+0000 mon.smithi053 (mon.0) 264 : audit [DBG] from='mgr.14169 172.21.15.53:0/1343570546' entity='mgr.smithi053.erbylj' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2020-07-24T18:11:44.828 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:44 smithi187 bash[12265]: audit 2020-07-24T18:11:44.284616+0000 mon.smithi053 (mon.0) 264 : audit [DBG] from='mgr.14169 172.21.15.53:0/1343570546' entity='mgr.smithi053.erbylj' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2020-07-24T18:11:45.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jul 24 18:11:45 smithi053 bash[12319]: cluster 2020-07-24T18:11:44.292523+0000 mgr.smithi053.erbylj (mgr.14169) 69 : cluster [DBG] pgmap v39: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2020-07-24T18:11:45.748 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:45 smithi187 bash[12265]: cluster 2020-07-24T18:11:44.292523+0000 mgr.smithi053.erbylj (mgr.14169) 69 : cluster [DBG] pgmap v39: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2020-07-24T18:11:46.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:46 smithi187 bash[12265]: cephadm 2020-07-24T18:11:45.601455+0000 mgr.smithi053.erbylj (mgr.14169) 70 : cephadm [ERR] cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploy daemon crash.smithi187 ... 2020-07-24T18:11:46.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:46 smithi187 bash[12265]: INFO:cephadm:Non-zero exit code 127 from /usr/bin/podman run --rm --net=host --ipc=host -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff -e NODE_NAME=smithi187 --entrypoint stat quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff -c %u %g /var/lib/ceph 2020-07-24T18:11:46.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:46 smithi187 bash[12265]: INFO:cephadm:stat:stderr Error: container_linux.go:349: starting container process caused "exec: \"stat\": executable file not found in $PATH": OCI runtime command not found error
Related issues
History
#1 Updated by Sebastian Wagner over 3 years ago
- Description updated (diff)
#2 Updated by Sebastian Wagner over 3 years ago
funny thing is: it was eventually deployed:
2020-07-24T18:15:41.726 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID 2020-07-24T18:15:41.726 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 running (3m) 74s ago 5m 0.20.0 docker.io/prom/alertmanager:v0.20.0 0881eb8f169f b323d85a7e14 2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053 smithi053 running (5m) 74s ago 5m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 943deeae7f0f 2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi187 smithi187 running (3m) 76s ago 3m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 e82a1d365979 2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053 smithi053 running (4m) 74s ago 4m 6.6.2 docker.io/ceph/ceph-grafana:6.6.2 a0dce381714a 6dd87560770a 2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.erbylj smithi053 running (6m) 74s ago 6m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 5e647371b21e 2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi187.sayfjr smithi187 running (3m) 76s ago 3m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 3a7af862b07b 2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053 smithi053 running (6m) 74s ago 6m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 a851a0174c0a 2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi187 smithi187 running (4m) 76s ago 4m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 212157f82e4f 2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 running (4m) 74s ago 4m 0.18.1 docker.io/prom/node-exporter:v0.18.1 e5a616e4b9cf 6a9e5b90a6c8 2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi187 smithi187 running (3m) 76s ago 3m 0.18.1 docker.io/prom/node-exporter:v0.18.1 e5a616e4b9cf 2511820d0f1d 2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (3m) 74s ago 3m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 4f8c603a608a 2020-07-24T18:15:41.729 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi053 running (3m) 74s ago 3m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 5120d6b4f625 2020-07-24T18:15:41.729 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi053 running (2m) 74s ago 2m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 888e18a6219b 2020-07-24T18:15:41.729 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (2m) 74s ago 2m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 cd9bd84f03f4 2020-07-24T18:15:41.729 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi187 running (2m) 76s ago 2m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 1a201c910a91 2020-07-24T18:15:41.730 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi187 running (112s) 76s ago 114s 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 e352455cc168 2020-07-24T18:15:41.730 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi187 running (94s) 76s ago 96s 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 5dab9bcc6712 2020-07-24T18:15:41.730 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi187 running (77s) 76s ago 79s 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff 34453d280d95 b1a5a18ebc04 2020-07-24T18:15:41.730 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053 smithi053 running (3m) 74s ago 4m 2.18.1 docker.io/prom/prometheus:v2.18.1 de242295e225 15f6c91c440b 2020
#3 Updated by Deepika Upadhyay over 3 years ago
2020-08-21T02:46:48.316 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: time="2020-08-21T02:46:48Z" level=error msg="unable to remove container 93488819ddcce8e2873eb8ab7665ce8761a15f308a2b5c05a9d579bb4983ed38 after failing to start and attach to it" 2020-08-21T02:46:48.367 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: Error: container_linux.go:345: starting container process caused "exec: \"/usr/bin/ceph-mon\": stat /usr/bin/ceph-mon: no such file or directory" 2020-08-21T02:46:48.368 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: : OCI runtime error
/a/yuriw-2020-08-20_00:20:21-rados-wip-yuri7-testing-2020-08-19-2051-octopus-distro-basic-smithi/5360912/teuthology.log
#4 Updated by Sebastian Wagner over 3 years ago
- Related to Bug #46529: cephadm: error removing storage for container "...-mon": remove /var/lib/containers/storage/overlay/.../merged: device or resource busy added
#5 Updated by Sebastian Wagner over 3 years ago
Deepika Upadhyay wrote:
[...]
/a/yuriw-2020-08-20_00:20:21-rados-wip-yuri7-testing-2020-08-19-2051-octopus-distro-basic-smithi/5360912/teuthology.log
Added to https://tracker.ceph.com/issues/46529#note-13
https://pulpito.ceph.com/yuriw-2020-08-20_19:48:15-rados-wip-yuri-testing-2020-08-17-1723-octopus-distro-basic-smithi/
5363117, 5362886, 5362916, 5362899, 5363031, 5363001
#6 Updated by Deepika Upadhyay over 3 years ago
Sebastian Wagner wrote:
Deepika Upadhyay wrote:
[...]
/a/yuriw-2020-08-20_00:20:21-rados-wip-yuri7-testing-2020-08-19-2051-octopus-distro-basic-smithi/5360912/teuthology.log
Added to https://tracker.ceph.com/issues/46529#note-13
https://pulpito.ceph.com/yuriw-2020-08-20_19:48:15-rados-wip-yuri-testing-2020-08-17-1723-octopus-distro-basic-smithi/
5363117, 5362886, 5362916, 5362899, 5363031, 5363001
looks specific to:
os_type: centos os_version: '7.6'
#7 Updated by Sebastian Wagner almost 3 years ago
- Status changed from New to Can't reproduce