Project

General

Profile

Bug #46704

container_linux.go:349: "exec: \"stat\": executable file not found

Added by Sebastian Wagner about 2 months ago. Updated 19 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
cephadm (binary)
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

Description: rados:cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start}

https://pulpito.ceph.com/swagner-2020-07-24_14:45:11-rados:cephadm-wip-swagner-testing-2020-07-24-1041-distro-basic-smithi/5253591/

2020-07-24T18:11:43.836 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:43 smithi187 bash[12265]: cluster 2020-07-24T18:11:43.342367+0000 mon.smithi053 (mon.0) 263 : cluster [WRN] overall HEALTH_WARN failed to probe daemons or devices; Reduced data availability: 1 pg inactive
2020-07-24T18:11:44.617 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jul 24 18:11:44 smithi053 bash[12319]: audit 2020-07-24T18:11:44.284616+0000 mon.smithi053 (mon.0) 264 : audit [DBG] from='mgr.14169 172.21.15.53:0/1343570546' entity='mgr.smithi053.erbylj' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch
2020-07-24T18:11:44.828 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:44 smithi187 bash[12265]: audit 2020-07-24T18:11:44.284616+0000 mon.smithi053 (mon.0) 264 : audit [DBG] from='mgr.14169 172.21.15.53:0/1343570546' entity='mgr.smithi053.erbylj' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch
2020-07-24T18:11:45.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jul 24 18:11:45 smithi053 bash[12319]: cluster 2020-07-24T18:11:44.292523+0000 mgr.smithi053.erbylj (mgr.14169) 69 : cluster [DBG] pgmap v39: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-07-24T18:11:45.748 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:45 smithi187 bash[12265]: cluster 2020-07-24T18:11:44.292523+0000 mgr.smithi053.erbylj (mgr.14169) 69 : cluster [DBG] pgmap v39: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-07-24T18:11:46.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:46 smithi187 bash[12265]: cephadm 2020-07-24T18:11:45.601455+0000 mgr.smithi053.erbylj (mgr.14169) 70 : cephadm [ERR] cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploy daemon crash.smithi187 ...
2020-07-24T18:11:46.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:46 smithi187 bash[12265]: INFO:cephadm:Non-zero exit code 127 from /usr/bin/podman run --rm --net=host --ipc=host -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff -e NODE_NAME=smithi187 --entrypoint stat quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff -c %u %g /var/lib/ceph
2020-07-24T18:11:46.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:46 smithi187 bash[12265]: INFO:cephadm:stat:stderr Error: container_linux.go:349: starting container process caused "exec: \"stat\": executable file not found in $PATH": OCI runtime command not found error

Related issues

Related to Orchestrator - Bug #46529: cephadm: error removing storage for container "...-mon": remove /var/lib/containers/storage/overlay/.../merged: device or resource busy Pending Backport

History

#1 Updated by Sebastian Wagner about 2 months ago

  • Description updated (diff)

#2 Updated by Sebastian Wagner about 2 months ago

funny thing is: it was eventually deployed:

2020-07-24T18:15:41.726 INFO:teuthology.orchestra.run.smithi053.stdout:NAME                     HOST       STATUS          REFRESHED  AGE   VERSION                   IMAGE NAME                                                          IMAGE ID      CONTAINER ID
2020-07-24T18:15:41.726 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053   smithi053  running (3m)    74s ago    5m    0.20.0                    docker.io/prom/alertmanager:v0.20.0                                 0881eb8f169f  b323d85a7e14
2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053          smithi053  running (5m)    74s ago    5m    16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  943deeae7f0f
2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi187          smithi187  running (3m)    76s ago    3m    16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  e82a1d365979
2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053        smithi053  running (4m)    74s ago    4m    6.6.2                     docker.io/ceph/ceph-grafana:6.6.2                                   a0dce381714a  6dd87560770a
2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.erbylj     smithi053  running (6m)    74s ago    6m    16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  5e647371b21e
2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi187.sayfjr     smithi187  running (3m)    76s ago    3m    16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  3a7af862b07b
2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053            smithi053  running (6m)    74s ago    6m    16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  a851a0174c0a
2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi187            smithi187  running (4m)    76s ago    4m    16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  212157f82e4f
2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053  smithi053  running (4m)    74s ago    4m    0.18.1                    docker.io/prom/node-exporter:v0.18.1                                e5a616e4b9cf  6a9e5b90a6c8
2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi187  smithi187  running (3m)    76s ago    3m    0.18.1                    docker.io/prom/node-exporter:v0.18.1                                e5a616e4b9cf  2511820d0f1d
2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0                    smithi053  running (3m)    74s ago    3m    16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  4f8c603a608a
2020-07-24T18:15:41.729 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1                    smithi053  running (3m)    74s ago    3m    16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  5120d6b4f625
2020-07-24T18:15:41.729 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2                    smithi053  running (2m)    74s ago    2m    16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  888e18a6219b
2020-07-24T18:15:41.729 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3                    smithi053  running (2m)    74s ago    2m    16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  cd9bd84f03f4
2020-07-24T18:15:41.729 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4                    smithi187  running (2m)    76s ago    2m    16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  1a201c910a91
2020-07-24T18:15:41.730 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5                    smithi187  running (112s)  76s ago    114s  16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  e352455cc168
2020-07-24T18:15:41.730 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6                    smithi187  running (94s)   76s ago    96s   16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  5dab9bcc6712
2020-07-24T18:15:41.730 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7                    smithi187  running (77s)   76s ago    79s   16.0.0-3805-gfa7e6da8476  quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff  34453d280d95  b1a5a18ebc04
2020-07-24T18:15:41.730 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053     smithi053  running (3m)    74s ago    4m    2.18.1                    docker.io/prom/prometheus:v2.18.1                                   de242295e225  15f6c91c440b
2020

#3 Updated by Deepika Upadhyay about 1 month ago

2020-08-21T02:46:48.316 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: time="2020-08-21T02:46:48Z" level=error msg="unable to remove container 93488819ddcce8e2873eb8ab7665ce8761a15f308a2b5c05a9d579bb4983ed38 after failing to start and attach to it" 
2020-08-21T02:46:48.367 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: Error: container_linux.go:345: starting container process caused "exec: \"/usr/bin/ceph-mon\": stat /usr/bin/ceph-mon: no such file or directory" 
2020-08-21T02:46:48.368 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: : OCI runtime error

/a/yuriw-2020-08-20_00:20:21-rados-wip-yuri7-testing-2020-08-19-2051-octopus-distro-basic-smithi/5360912/teuthology.log

#4 Updated by Sebastian Wagner about 1 month ago

  • Related to Bug #46529: cephadm: error removing storage for container "...-mon": remove /var/lib/containers/storage/overlay/.../merged: device or resource busy added

#5 Updated by Sebastian Wagner about 1 month ago

Deepika Upadhyay wrote:

[...]

/a/yuriw-2020-08-20_00:20:21-rados-wip-yuri7-testing-2020-08-19-2051-octopus-distro-basic-smithi/5360912/teuthology.log

Added to https://tracker.ceph.com/issues/46529#note-13

https://pulpito.ceph.com/yuriw-2020-08-20_19:48:15-rados-wip-yuri-testing-2020-08-17-1723-octopus-distro-basic-smithi/
5363117, 5362886, 5362916, 5362899, 5363031, 5363001

#6 Updated by Deepika Upadhyay 19 days ago

Sebastian Wagner wrote:

Deepika Upadhyay wrote:

[...]

/a/yuriw-2020-08-20_00:20:21-rados-wip-yuri7-testing-2020-08-19-2051-octopus-distro-basic-smithi/5360912/teuthology.log

Added to https://tracker.ceph.com/issues/46529#note-13

https://pulpito.ceph.com/yuriw-2020-08-20_19:48:15-rados-wip-yuri-testing-2020-08-17-1723-octopus-distro-basic-smithi/
5363117, 5362886, 5362916, 5362899, 5363031, 5363001

looks specific to:

 os_type: centos
  os_version: '7.6'

Also available in: Atom PDF