https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2020-04-08T09:35:23ZCeph Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1627812020-04-08T09:35:23ZSebastian Wagner
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-6 priority-high2 closed" href="/issues/44777">Bug #44777</a>: podman: stat /usr/bin/ceph-mon: no such file or directory, then unable to remove container</i> added</li></ul> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1642832020-04-27T21:55:22ZNeha Ojhanojha@redhat.com
<ul></ul><p>/a/yuriw-2020-04-25_15:46:30-rados-wip-yuri4-testing-2020-04-25-0009-master-distro-basic-smithi/4984285</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1650982020-05-06T18:22:29ZJosh Durgin
<ul></ul><p>Could this be caused by the container image not being built yet, or would that present as a different error? With any cephadm jobs teuthology-suite could check that the container image is present just like it does for packages to avoid that problem.</p>
<p>This occurred again in a suite scheduled shortly after package builds finished:</p>
<p>/a/joshd-2020-05-06_08:23:52-rados-wip-joshd-fix-octopus-distro-basic-smithi/5028167/</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1651582020-05-07T11:30:40ZSebastian Wagner
<ul><li><strong>Duplicated by</strong> <i><a class="issue tracker-1 status-10 priority-5 priority-high3 closed" href="/issues/45421">Bug #45421</a>: cephadm: MaxWhileTries: Waiting for 3 mons in monmap: "unable to remove container c3ed65093dd89d593e40d2d1bbfa03c8dcb5f53ba7bdda77eacde8d9f1a9c28e after failing to start and attach to it"</i> added</li></ul> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1651612020-05-07T11:32:40ZSebastian Wagner
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>Sebastian Wagner</i></li></ul> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1651622020-05-07T11:35:45ZSebastian Wagner
<ul></ul><p>In the past, <a class="external" href="https://github.com/ceph/ceph/pull/34091">https://github.com/ceph/ceph/pull/34091</a> was able to reproduce this bug consistently. I'll look into resurrecting it. Let's see if it provides a way to find cause.</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1651972020-05-07T15:05:26ZSebastian Wagner
<ul><li><strong>Subject</strong> changed from <i>octopus: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directory</i> to <i>cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directory</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul><p><a class="external" href="http://pulpito.ceph.com/mgfritch-2020-05-07_02:27:06-rados-wip-mgfritch-testing-2020-05-06-1821-distro-basic-smithi/5029062">http://pulpito.ceph.com/mgfritch-2020-05-07_02:27:06-rados-wip-mgfritch-testing-2020-05-06-1821-distro-basic-smithi/5029062</a></p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1656672020-05-12T11:35:03ZSebastian Wagner
<ul></ul><pre>
2020-05-06T10:25:58.126 INFO:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph-ci/ceph:0b54056fb60dceb5086e11269bac7b044d365904 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e6313c4-8f83-11
2020-05-06T10:25:58.182 INFO:ceph.mon.a.smithi131.stdout:May 06 10:25:58 smithi131 bash[10137]: cluster 2020-05-06T10:25:57.866815+0000 mgr.y (mgr.14141) 54 : cluster [DBG] pgmap v45: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-06T10:25:58.184 INFO:ceph.mon.b.smithi053.stdout:-- Logs begin at Wed 2020-05-06 10:14:19 UTC. --
2020-05-06T10:25:58.184 INFO:ceph.mon.b.smithi053.stdout:May 06 10:25:57 smithi053 podman[10178]: 2020-05-06 10:25:57.675609957 +0000 UTC m=+0.551746053 container create 3e6b87c9529fcad64f1dd6c148c7f8944a4b52d8f2e63d8ce2b331ef06d177a0 (image=quay.io/ceph-ci/ceph
2020-05-06T10:25:59.992 INFO:ceph.mon.a.smithi131.stdout:May 06 10:25:59 smithi131 bash[10137]: cluster 2020-05-06T10:25:59.867300+0000 mgr.y (mgr.14141) 55 : cluster [DBG] pgmap v46: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-06T10:26:01.991 INFO:ceph.mon.a.smithi131.stdout:May 06 10:26:01 smithi131 bash[10137]: cluster 2020-05-06T10:26:01.867801+0000 mgr.y (mgr.14141) 56 : cluster [DBG] pgmap v47: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-06T10:26:03.991 INFO:ceph.mon.a.smithi131.stdout:May 06 10:26:03 smithi131 bash[10137]: cluster 2020-05-06T10:26:03.868280+0000 mgr.y (mgr.14141) 57 : cluster [DBG] pgmap v48: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-06T10:26:05.722 INFO:ceph.mon.b.smithi053.stdout:May 06 10:26:05 smithi053 podman[10178]: 2020-05-06 10:26:05.72028671 +0000 UTC m=+8.596422879 container remove 3e6b87c9529fcad64f1dd6c148c7f8944a4b52d8f2e63d8ce2b331ef06d177a0 (image=quay.io/ceph-ci/ceph:
2020-05-06T10:26:05.725 INFO:ceph.mon.b.smithi053.stdout:May 06 10:26:05 smithi053 bash[10174]: time="2020-05-06T10:26:05Z" level=error msg="unable to remove container 3e6b87c9529fcad64f1dd6c148c7f8944a4b52d8f2e63d8ce2b331ef06d177a0 after failing to start and at
2020-05-06T10:26:05.812 INFO:ceph.mon.b.smithi053.stdout:May 06 10:26:05 smithi053 bash[10174]: Error: container_linux.go:345: starting container process caused "exec: \"/usr/bin/ceph-mon\": stat /usr/bin/ceph-mon: no such file or directory"
2020-05-06T10:26:05.812 INFO:ceph.mon.b.smithi053.stdout:May 06 10:26:05 smithi053 bash[10174]: : OCI runtime error
2020-05-06T10:26:05.829 INFO:ceph.mon.b.smithi053.stdout:May 06 10:26:05 smithi053 systemd[1]: ceph-8e6313c4-8f83-11ea-a068-001a4aab830c@mon.b.service: main process exited, code=exited, status=127/n/a
2020-05-06T10:26:05.988 INFO:ceph.mon.b.smithi053.stdout:May 06 10:26:05 smithi053 podman[10437]: Error: no container with name or ID ceph-8e6313c4-8f83-11ea-a068-001a4aab830c-mon.b found: no such container
</pre> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1656682020-05-12T11:36:18ZSebastian Wagner
<ul></ul><pre>
2020-04-25T21:57:55.424 INFO:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph-ci/ceph:fec0296dffa2b7ab61f520f54a835a822a2b2fa4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5a63e2d4-873f-11ea-a068-001a4aab830c -- ceph mon dump -f json
2020-04-25T21:57:55.481 INFO:ceph.mon.b.smithi033.stdout:-- Logs begin at Sat 2020-04-25 21:46:39 UTC. --
2020-04-25T21:57:55.481 INFO:ceph.mon.b.smithi033.stdout:Apr 25 21:57:55 smithi033 podman[8415]: 2020-04-25 21:57:55.001506419 +0000 UTC m=+0.656050814 container create 94571bdfe3f9b8076522bfa1d95d5424f080af4ddd876fc8d0d41677ba25b638 (image=quay.io/ceph-ci/ceph:fec0296dffa2b7ab61f520f54a835a822a2b2fa4, name=ceph-5a63e2d4-873f-11ea-a068-001a4aab830c-mon.b)
2020-04-25T21:57:55.619 INFO:ceph.mon.a.smithi114.stdout:Apr 25 21:57:55 smithi114 bash[8807]: cluster 2020-04-25T21:57:54.376174+0000 mgr.y (mgr.14143) 62 : cluster [DBG] pgmap v53: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-04-25T21:57:55.623 INFO:ceph.mon.c.smithi042.stdout:Apr 25 21:57:55 smithi042 bash[8897]: cluster 2020-04-25T21:57:54.376174+0000 mgr.y (mgr.14143) 62 : cluster [DBG] pgmap v53: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-04-25T21:57:56.627 INFO:ceph.mon.a.smithi114.stdout:Apr 25 21:57:56 smithi114 bash[8807]: cluster 2020-04-25T21:57:56.376626+0000 mgr.y (mgr.14143) 63 : cluster [DBG] pgmap v54: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-04-25T21:57:56.628 INFO:ceph.mon.c.smithi042.stdout:Apr 25 21:57:56 smithi042 bash[8897]: cluster 2020-04-25T21:57:56.376626+0000 mgr.y (mgr.14143) 63 : cluster [DBG] pgmap v54: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-04-25T21:57:59.454 INFO:ceph.mon.a.smithi114.stdout:Apr 25 21:57:59 smithi114 bash[8807]: cluster 2020-04-25T21:57:58.377092+0000 mgr.y (mgr.14143) 64 : cluster [DBG] pgmap v55: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-04-25T21:57:59.455 INFO:ceph.mon.c.smithi042.stdout:Apr 25 21:57:59 smithi042 bash[8897]: cluster 2020-04-25T21:57:58.377092+0000 mgr.y (mgr.14143) 64 : cluster [DBG] pgmap v55: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-04-25T21:58:01.237 INFO:ceph.mon.a.smithi114.stdout:Apr 25 21:58:01 smithi114 bash[8807]: cluster 2020-04-25T21:58:00.377556+0000 mgr.y (mgr.14143) 65 : cluster [DBG] pgmap v56: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-04-25T21:58:01.238 INFO:ceph.mon.c.smithi042.stdout:Apr 25 21:58:01 smithi042 bash[8897]: cluster 2020-04-25T21:58:00.377556+0000 mgr.y (mgr.14143) 65 : cluster [DBG] pgmap v56: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-04-25T21:58:03.023 INFO:ceph.mon.b.smithi033.stdout:Apr 25 21:58:03 smithi033 podman[8415]: 2020-04-25 21:58:03.023594912 +0000 UTC m=+8.678139524 container remove 94571bdfe3f9b8076522bfa1d95d5424f080af4ddd876fc8d0d41677ba25b638 (image=quay.io/ceph-ci/ceph:fec0296dffa2b7ab61f520f54a835a822a2b2fa4, name=ceph-5a63e2d4-873f-11ea-a068-001a4aab830c-mon.b)
2020-04-25T21:58:03.026 INFO:ceph.mon.b.smithi033.stdout:Apr 25 21:58:03 smithi033 bash[8411]: time="2020-04-25T21:58:03Z" level=error msg="unable to remove container 94571bdfe3f9b8076522bfa1d95d5424f080af4ddd876fc8d0d41677ba25b638 after failing to start and attach to it"
2020-04-25T21:58:03.094 INFO:ceph.mon.b.smithi033.stdout:Apr 25 21:58:03 smithi033 bash[8411]: Error: container_linux.go:345: starting container process caused "exec: \"/usr/bin/ceph-mon\": stat /usr/bin/ceph-mon: no such file or directory"
2020-04-25T21:58:03.094 INFO:ceph.mon.b.smithi033.stdout:Apr 25 21:58:03 smithi033 bash[8411]: : OCI runtime error
2020-04-25T21:58:03.113 INFO:ceph.mon.b.smithi033.stdout:Apr 25 21:58:03 smithi033 systemd[1]: ceph-5a63e2d4-873f-11ea-a068-001a4aab830c@mon.b.service: main process exited, code=exited, status=127/n/a
2020-04-25T21:58:03.293 INFO:ceph.mon.b.smithi033.stdout:Apr 25 21:58:03 smithi033 podman[8685]: Error: no container with name or ID ceph-5a63e2d4-873f-11ea-a068-001a4aab830c-mon.b found: no such container
2020-04-25T21:58:03.314 INFO:ceph.mon.b.smithi033.stdout:Apr 25 21:58:03 smithi033 systemd[1]: Unit ceph-5a63e2d4-873f-11ea-a068-001a4aab830c@mon.b.service entered failed state.
2020-04-25T21:58:03.315 INFO:ceph.mon.b.smithi033.stdout:Apr 25 21:58:03 smithi033 systemd[1]: ceph-5a63e2d4-873f-11ea-a068-001a4aab830c@mon.b.service failed.
</pre> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1656692020-05-12T11:39:48ZSebastian Wagner
<ul></ul><pre>
2020-05-08T15:44:31.009 INFO:tasks.cephadm:Waiting for 3 mons in monmap...
2020-05-08T15:44:31.010 INFO:teuthology.orchestra.run.smithi096:> true
2020-05-08T15:44:31.094 INFO:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph-ci/ceph:2e1f29558886bd90d7f04e6fced16b4a9464840b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7792d16e-9142-11ea-a068-001a4aab830c -- ceph mon dump -f json
2020-05-08T15:44:31.148 INFO:ceph.mon.b.smithi096.stdout:-- Logs begin at Fri 2020-05-08 15:36:32 UTC. --
2020-05-08T15:44:31.148 INFO:ceph.mon.b.smithi096.stdout:May 08 15:44:30 smithi096 podman[6877]: 2020-05-08 15:44:30.680006536 +0000 UTC m=+0.442275939 container create cb703f4fa7c417c57ecaa89052f59755a6115b0b29dc5c70b703789fefc6f612 (image=quay.io/ceph-ci/ceph:2e1f29558886bd90d7f04e6fced16b4a9464840b, name=ceph-7792d16e-9142-11ea-a068-001a4aab830c-mon.b)
2020-05-08T15:44:32.240 INFO:ceph.mon.a.smithi196.stdout:May 08 15:44:32 smithi196 bash[6946]: cluster 2020-05-08T15:44:30.628739+0000 mgr.y (mgr.14142) 35 : cluster [DBG] pgmap v29: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-08T15:44:32.241 INFO:ceph.mon.c.smithi196.stdout:May 08 15:44:32 smithi196 bash[11439]: cluster 2020-05-08T15:44:30.628739+0000 mgr.y (mgr.14142) 35 : cluster [DBG] pgmap v29: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-08T15:44:34.244 INFO:ceph.mon.a.smithi196.stdout:May 08 15:44:34 smithi196 bash[6946]: cluster 2020-05-08T15:44:32.629261+0000 mgr.y (mgr.14142) 36 : cluster [DBG] pgmap v30: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-08T15:44:34.245 INFO:ceph.mon.c.smithi196.stdout:May 08 15:44:34 smithi196 bash[11439]: cluster 2020-05-08T15:44:32.629261+0000 mgr.y (mgr.14142) 36 : cluster [DBG] pgmap v30: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-08T15:44:36.248 INFO:ceph.mon.a.smithi196.stdout:May 08 15:44:36 smithi196 bash[6946]: cluster 2020-05-08T15:44:34.629756+0000 mgr.y (mgr.14142) 37 : cluster [DBG] pgmap v31: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-08T15:44:36.249 INFO:ceph.mon.c.smithi196.stdout:May 08 15:44:36 smithi196 bash[11439]: cluster 2020-05-08T15:44:34.629756+0000 mgr.y (mgr.14142) 37 : cluster [DBG] pgmap v31: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-08T15:44:38.253 INFO:ceph.mon.a.smithi196.stdout:May 08 15:44:38 smithi196 bash[6946]: cluster 2020-05-08T15:44:36.630260+0000 mgr.y (mgr.14142) 38 : cluster [DBG] pgmap v32: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-08T15:44:38.253 INFO:ceph.mon.c.smithi196.stdout:May 08 15:44:38 smithi196 bash[11439]: cluster 2020-05-08T15:44:36.630260+0000 mgr.y (mgr.14142) 38 : cluster [DBG] pgmap v32: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-08T15:44:38.615 INFO:ceph.mon.b.smithi096.stdout:May 08 15:44:38 smithi096 podman[6877]: 2020-05-08 15:44:38.61445606 +0000 UTC m=+8.376725570 container remove cb703f4fa7c417c57ecaa89052f59755a6115b0b29dc5c70b703789fefc6f612 (image=quay.io/ceph-ci/ceph:2e1f29558886bd90d7f04e6fced16b4a9464840b, name=ceph-7792d16e-9142-11ea-a068-001a4aab830c-mon.b)
2020-05-08T15:44:38.618 INFO:ceph.mon.b.smithi096.stdout:May 08 15:44:38 smithi096 bash[6873]: time="2020-05-08T15:44:38Z" level=error msg="unable to remove container cb703f4fa7c417c57ecaa89052f59755a6115b0b29dc5c70b703789fefc6f612 after failing to start and attach to it"
2020-05-08T15:44:38.674 INFO:ceph.mon.b.smithi096.stdout:May 08 15:44:38 smithi096 bash[6873]: Error: container_linux.go:345: starting container process caused "exec: \"/usr/bin/ceph-mon\": stat /usr/bin/ceph-mon: no such file or directory"
2020-05-08T15:44:38.675 INFO:ceph.mon.b.smithi096.stdout:May 08 15:44:38 smithi096 bash[6873]: : OCI runtime error
2020-05-08T15:44:38.690 INFO:ceph.mon.b.smithi096.stdout:May 08 15:44:38 smithi096 systemd[1]: ceph-7792d16e-9142-11ea-a068-001a4aab830c@mon.b.service: main process exited, code=exited, status=127/n/a
2020-05-08T15:44:38.816 INFO:ceph.mon.b.smithi096.stdout:May 08 15:44:38 smithi096 podman[7136]: Error: no container with name or ID ceph-7792d16e-9142-11ea-a068-001a4aab830c-mon.b found: no such container
</pre> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1656742020-05-12T12:08:31ZSebastian Wagner
<ul></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/35018">https://github.com/ceph/ceph/pull/35018</a> might make this thing go away, without fixing the underlying issue.</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1661402020-05-19T12:12:22ZSebastian Wagner
<ul></ul><p>seems that I'm close to unable to reproduce this reliably.</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1662752020-05-21T00:43:08ZNeha Ojhanojha@redhat.com
<ul></ul><p>Looks similar</p>
<pre>
2020-05-20T21:17:30.996 INFO:teuthology.orchestra.run.smithi038:mon.b> sudo journalctl -f -n 0 -u ceph-e9ddf488-9ade-11ea-a06a-001a4aab830c@mon.b.service
2020-05-20T21:17:31.002 INFO:tasks.cephadm:Waiting for 2 mons in monmap...
2020-05-20T21:17:31.002 INFO:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph-ci/ceph:2e8572496bd5b6aa1de9ccadd3e0d171c60aee81 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e9ddf488-9ade-11ea-a06a-001a4aab830c -- ceph mon dump -f json
2020-05-20T21:17:31.138 INFO:ceph.mon.b.smithi038.stdout:-- Logs begin at Wed 2020-05-20 21:06:47 UTC. --
2020-05-20T21:17:31.138 INFO:ceph.mon.b.smithi038.stdout:May 20 21:17:30 smithi038 podman[7373]: 2020-05-20 21:17:30.741649334 +0000 UTC m=+0.506565325 container create ec7f880ef9761d95f33f95bba292813602748e57661d56298f70f5a349ed5e86 (image=quay.io/ceph-ci/ceph:2e8572496bd5b6aa1de9ccadd3e0d171c60aee81, name=ceph-e9ddf488-9ade-11ea-a06a-001a4aab830c-mon.b)
2020-05-20T21:17:31.871 INFO:ceph.mon.a.smithi185.stdout:May 20 21:17:31 smithi185 bash[7587]: cluster 2020-05-20T21:17:31.778601+0000 mgr.y (mgr.14140) 57 : cluster [DBG] pgmap v44: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-20T21:17:33.863 INFO:ceph.mon.a.smithi185.stdout:May 20 21:17:33 smithi185 bash[7587]: cluster 2020-05-20T21:17:33.779080+0000 mgr.y (mgr.14140) 58 : cluster [DBG] pgmap v45: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-20T21:17:36.638 INFO:ceph.mon.a.smithi185.stdout:May 20 21:17:36 smithi185 bash[7587]: cluster 2020-05-20T21:17:35.779516+0000 mgr.y (mgr.14140) 59 : cluster [DBG] pgmap v46: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-20T21:17:37.866 INFO:ceph.mon.a.smithi185.stdout:May 20 21:17:37 smithi185 bash[7587]: cluster 2020-05-20T21:17:37.780155+0000 mgr.y (mgr.14140) 60 : cluster [DBG] pgmap v47: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-20T21:17:38.831 INFO:ceph.mon.b.smithi038.stdout:May 20 21:17:38 smithi038 podman[7373]: 2020-05-20 21:17:38.829952419 +0000 UTC m=+8.594868476 container remove ec7f880ef9761d95f33f95bba292813602748e57661d56298f70f5a349ed5e86 (image=quay.io/ceph-ci/ceph:2e8572496bd5b6aa1de9ccadd3e0d171c60aee81, name=ceph-e9ddf488-9ade-11ea-a06a-001a4aab830c-mon.b)
2020-05-20T21:17:38.834 INFO:ceph.mon.b.smithi038.stdout:May 20 21:17:38 smithi038 bash[7369]: time="2020-05-20T21:17:38Z" level=error msg="unable to remove container ec7f880ef9761d95f33f95bba292813602748e57661d56298f70f5a349ed5e86 after failing to start and attach to it"
2020-05-20T21:17:38.952 INFO:ceph.mon.b.smithi038.stdout:May 20 21:17:38 smithi038 bash[7369]: Error: container_linux.go:345: starting container process caused "exec: \"/usr/bin/ceph-mon\": stat /usr/bin/ceph-mon: no such file or directory"
</pre>
<p>/a/nojha-2020-05-20_19:45:21-rados-master-distro-basic-smithi/5073466 - this is on latest master</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1664382020-05-22T02:54:13ZKefu Chaitchaikov@gmail.com
<ul></ul><pre>
2020-05-21T11:22:56.079 INFO:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph-ci/ceph:62b4f5047ac335093fb47a2897524ad3f1e6aa9d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 04030f80-9b55-11ea-a06a-001a4aab830c -- ceph mon dump -f json
2020-05-21T11:22:57.724 INFO:ceph.mon.a.smithi185.stdout:May 21 11:22:57 smithi185 bash[8583]: cluster 2020-05-21T11:22:56.149439+0000 mgr.y (mgr.14141) 56 : cluster [DBG] pgmap v43: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-21T11:22:59.726 INFO:ceph.mon.a.smithi185.stdout:May 21 11:22:59 smithi185 bash[8583]: cluster 2020-05-21T11:22:58.149920+0000 mgr.y (mgr.14141) 57 : cluster [DBG] pgmap v44: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-21T11:23:01.729 INFO:ceph.mon.a.smithi185.stdout:May 21 11:23:01 smithi185 bash[8583]: cluster 2020-05-21T11:23:00.150408+0000 mgr.y (mgr.14141) 58 : cluster [DBG] pgmap v45: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-21T11:23:03.730 INFO:ceph.mon.a.smithi185.stdout:May 21 11:23:03 smithi185 bash[8583]: cluster 2020-05-21T11:23:02.150880+0000 mgr.y (mgr.14141) 59 : cluster [DBG] pgmap v46: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
2020-05-21T11:23:03.877 INFO:ceph.mon.b.smithi084.stdout:-- Logs begin at Thu 2020-05-21 11:11:16 UTC. --
2020-05-21T11:23:03.877 INFO:ceph.mon.b.smithi084.stdout:May 21 11:22:55 smithi084 podman[8564]: 2020-05-21 11:22:55.830665394 +0000 UTC m=+0.467398441 container create 0f4c2860764cf01c0ba1fd566a971919b3c32d31edd9810cb52021623717cf10 (image=quay.io/ceph-ci/ceph:62b4f5047ac335093fb47a2897524ad3f1e6aa9d, name=ceph-04030f80-9b55-11ea-a06a-001a4aab830c-mon.b)
2020-05-21T11:23:03.878 INFO:ceph.mon.b.smithi084.stdout:May 21 11:23:03 smithi084 podman[8564]: 2020-05-21 11:23:03.874235726 +0000 UTC m=+8.510968806 container remove 0f4c2860764cf01c0ba1fd566a971919b3c32d31edd9810cb52021623717cf10 (image=quay.io/ceph-ci/ceph:62b4f5047ac335093fb47a2897524ad3f1e6aa9d, name=ceph-04030f80-9b55-11ea-a06a-001a4aab830c-mon.b)
2020-05-21T11:23:03.879 INFO:ceph.mon.b.smithi084.stdout:May 21 11:23:03 smithi084 bash[8559]: time="2020-05-21T11:23:03Z" level=error msg="unable to remove container 0f4c2860764cf01c0ba1fd566a971919b3c32d31edd9810cb52021623717cf10 after failing to start and attach to it"
2020-05-21T11:23:03.931 INFO:ceph.mon.b.smithi084.stdout:May 21 11:23:03 smithi084 bash[8559]: Error: container_linux.go:345: starting container process caused "exec: \"/usr/bin/ceph-mon\": stat /usr/bin/ceph-mon: no such file or directory"
2020-05-21T11:23:03.931 INFO:ceph.mon.b.smithi084.stdout:May 21 11:23:03 smithi084 bash[8559]: : OCI runtime error
2020-05-21T11:23:03.944 INFO:ceph.mon.b.smithi084.stdout:May 21 11:23:03 smithi084 systemd[1]: ceph-04030f80-9b55-11ea-a06a-001a4aab830c@mon.b.service: main process exited, code=exited, status=127/n/a
2020-05-21T11:23:04.115 INFO:ceph.mon.b.smithi084.stdout:May 21 11:23:04 smithi084 podman[8769]: Error: no container with name or ID ceph-04030f80-9b55-11ea-a06a-001a4aab830c-mon.b found: no such container
2020-05-21T11:23:04.130 INFO:ceph.mon.b.smithi084.stdout:May 21 11:23:04 smithi084 systemd[1]: Unit ceph-04030f80-9b55-11ea-a06a-001a4aab830c@mon.b.service entered failed state.
2020-05-21T11:23:04.131 INFO:ceph.mon.b.smithi084.stdout:May 21 11:23:04 smithi084 systemd[1]: ceph-04030f80-9b55-11ea-a06a-001a4aab830c@mon.b.service failed.
</pre>
<p>failed to add the second monitor</p>
<p>/a/kchai-2020-05-21_10:34:02-rados-wip-kefu-testing-2020-05-21-1652-distro-basic-smithi/5076314</p>
<p>please note, this issue also impacts the upgrade test.</p>
<pre>
rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/nautilus-v2only.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_7.6.yaml} msgr-failures/fastclose.yaml rados.yaml thrashers/pggrow.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml}
</pre>
<p>where we are using cephadm for deploying ceph monitors. while the error message looks like</p>
<blockquote>
<p>reached maximum tries (180) after waiting for 180 seconds</p>
</blockquote>
<p>see <a class="external" href="http://pulpito.ceph.com/kchai-2020-05-21_10:34:02-rados-wip-kefu-testing-2020-05-21-1652-distro-basic-smithi/5076314/">http://pulpito.ceph.com/kchai-2020-05-21_10:34:02-rados-wip-kefu-testing-2020-05-21-1652-distro-basic-smithi/5076314/</a></p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1664392020-05-22T02:54:56ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Priority</strong> changed from <i>High</i> to <i>Urgent</i></li></ul> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1667382020-05-27T02:45:43ZBrad Hubbardbhubbard@redhat.com
<ul></ul><p>/a/yuriw-2020-05-22_19:55:53-rados-wip-yuri-master_5.22.20-distro-basic-smithi/5083157<br />/a/yuriw-2020-05-22_19:55:53-rados-wip-yuri-master_5.22.20-distro-basic-smithi/5083387<br />/a/yuriw-2020-05-22_19:55:53-rados-wip-yuri-master_5.22.20-distro-basic-smithi/5083421<br />/a/yuriw-2020-05-22_19:55:53-rados-wip-yuri-master_5.22.20-distro-basic-smithi/5083521<br />/a/yuriw-2020-05-22_19:55:53-rados-wip-yuri-master_5.22.20-distro-basic-smithi/5083323</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1669232020-05-29T04:29:57ZBrad Hubbardbhubbard@redhat.com
<ul></ul><p>/a/yuriw-2020-05-28_02:23:45-rados-wip-yuri-master_5.27.20-distro-basic-smithi/5098059<br />/a/yuriw-2020-05-28_02:23:45-rados-wip-yuri-master_5.27.20-distro-basic-smithi/5097857<br />/a/yuriw-2020-05-28_02:23:45-rados-wip-yuri-master_5.27.20-distro-basic-smithi/5097925<br />/a/yuriw-2020-05-28_02:23:45-rados-wip-yuri-master_5.27.20-distro-basic-smithi/5097958<br />/a/yuriw-2020-05-28_02:23:45-rados-wip-yuri-master_5.27.20-distro-basic-smithi/5097827<br />/a/yuriw-2020-05-28_02:23:45-rados-wip-yuri-master_5.27.20-distro-basic-smithi/5097692</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1669262020-05-29T09:09:25ZSebastian Wagner
<ul></ul><p>maybe we can actually fix this by moving to a our internal registry</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1678732020-06-09T05:34:16ZKefu Chaitchaikov@gmail.com
<ul></ul><p>/a/kchai-2020-06-08_10:56:36-rados-wip-kefu-testing-2020-06-08-1713-distro-basic-smithi/5128793/</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1678862020-06-09T08:30:13ZSebastian Wagner
<ul></ul><p>fascinating:</p>
<pre>
systemd[1]: ceph-36889d04-a982-11ea-a06d-001a4aab830c@mon.b.service holdoff time over, scheduling restart.
systemd[1]: Stopped Ceph mon.b for 36889d04-a982-11ea-a06d-001a4aab830c.
systemd[1]: Starting Ceph mon.b for 36889d04-a982-11ea-a06d-001a4aab830c...
podman[9425]: Error: no container with name or ID ceph-36889d04-a982-11ea-a06d-001a4aab830c-mon.b found: no such container
systemd[1]: Started Ceph mon.b for 36889d04-a982-11ea-a06d-001a4aab830c.
bash[9449]: Error: no container with name or ID ceph-36889d04-a982-11ea-a06d-001a4aab830c-mon.b found: no such container
bash[9449]: Error: error creating container storage: the container name "ceph-36889d04-a982-11ea-a06d-001a4aab830c-mon.b" is already in use by "f265ae83cb9ed32b7ed6fd6e62a2e764549
systemd[1]: ceph-36889d04-a982-11ea-a06d-001a4aab830c@mon.b.service: main process exited, code=exited, status=125/n/a
podman[9489]: Error: no container with name or ID ceph-36889d04-a982-11ea-a06d-001a4aab830c-mon.b found: no such container
systemd[1]: Unit ceph-36889d04-a982-11ea-a06d-001a4aab830c@mon.b.service entered failed state.
systemd[1]: ceph-36889d04-a982-11ea-a06d-001a4aab830c@mon.b.service failed.
</pre> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1679802020-06-10T12:29:53ZSebastian Wagner
<ul></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/35524">https://github.com/ceph/ceph/pull/35524</a></p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1683042020-06-17T07:37:29ZSebastian Wagner
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-5 priority-high3 closed" href="/issues/46036">Bug #46036</a>: cephadm: killmode=none: systemd units failed, but containers still running</i> added</li></ul> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1693052020-06-29T20:03:31ZNeha Ojhanojha@redhat.com
<ul></ul><p>/a/yuriw-2020-06-29_16:59:21-rados-octopus-distro-basic-smithi/5189862</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1693222020-06-30T08:03:47ZSebastian Wagner
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Pending Backport</i></li></ul><p>was fixed in master.</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1693232020-06-30T08:04:24ZSebastian Wagner
<ul><li><strong>Pull request ID</strong> set to <i>35524</i></li></ul> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1704912020-07-13T21:19:25ZNeha Ojhanojha@redhat.com
<ul></ul><p>Sebastian, I am seeing similar failures in rados/thrash-old-clients on recent master, can you please confirm if they need a different tracker issue?</p>
<p>/a/teuthology-2020-07-12_07:01:02-rados-master-distro-basic-smithi/5217488<br />/a/teuthology-2020-07-12_07:01:02-rados-master-distro-basic-smithi/5217586</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1705242020-07-14T10:26:08ZSebastian Wagner
<ul></ul><p>yep, that's a different issue:</p>
<pre>
2020-07-12T15:39:57.736 INFO:journalctl@ceph.mon.c.smithi145.stdout:Jul 12 15:39:57 smithi145 bash[9707]: Error: error removing storage for container "ceph-567f836e-c455-11ea-a06e-001a4aab830c-mon.c": remove /var/lib/containers/storage/overlay/3e200c9a65162f9c55a682cb3ea6b559b07a6569a910eb1057ea4f995067f9eb/merged: device or resource busy
2020-07-12T15:39:58.066 INFO:journalctl@ceph.mon.c.smithi145.stdout:Jul 12 15:39:58 smithi145 bash[9707]: Error: error creating container storage: the container name "ceph-567f836e-c455-11ea-a06e-001a4aab830c-mon.c" is already in use by "4588abec9d4e9d178e083064e81a5766f0e2d800170592bc51a2d31f247e09b5". You have to remove that container to be able to reuse that name.: that name is already in use
</pre>
<p><a class="external" href="https://tracker.ceph.com/issues/46529">https://tracker.ceph.com/issues/46529</a></p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1705252020-07-14T10:39:33ZSebastian Wagner
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-6 priority-high2 closed" href="/issues/46529">Bug #46529</a>: cephadm: error removing storage for container "...-mon": remove /var/lib/containers/storage/overlay/.../merged: device or resource busy</i> added</li></ul> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1707742020-07-17T09:44:48ZSebastian Wagner
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1710912020-07-19T05:43:55ZKefu Chaitchaikov@gmail.com
<ul></ul><pre>
2020-07-18T17:09:58.444 INFO:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:a2848ece1471a0772679dea9aa70bd344a7f2a0b shell -c /etc/ceph/ceph.conf
-k /etc/ceph/ceph.client.admin.keyring --fsid f930da78-c918-11ea-a06f-001a4aab830c -- ceph mon dump -f json
2020-07-18T17:09:58.576 INFO:journalctl@ceph.mon.b.smithi070.stdout:-- Logs begin at Sat 2020-07-18 16:57:47 UTC. --
2020-07-18T17:09:58.579 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 18 17:09:58 smithi070 podman[9257]: 2020-07-18 17:09:58.220642722 +0000 UTC m=+0.474545854 container create f49dfa4f6185b3edbc4b028e286efb13e317e9349413c6a335fd2af75118815f (image=quay.ceph.io/ceph-ci/ceph:a2848ece1471a0772679dea9aa70bd344a7f2a0b, name=ceph-f930da78-c918-11ea-a06f-001a4aab830c-mon.b)
...
2020-07-18T17:10:06.155 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 18 17:10:06 smithi070 podman[9257]: 2020-07-18 17:10:06.159800433 +0000 UTC m=+8.413703562 container remove f49dfa4f6185b3edbc4b028e286efb13e317e9349413c6a335fd2af75118815f (image=quay.ceph.io/ceph-ci/ceph:a2848ece1471a0772679dea9aa70bd344a7f2a0b, name=ceph-f930da78-c918-11ea-a06f-001a4aab830c-mon.b)
2020-07-18T17:10:06.157 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 18 17:10:06 smithi070 bash[9216]: time="2020-07-18T17:10:06Z" level=error msg="unable to remove container f49dfa4f6185b3edbc4b028e286efb13e317e9349413c6a335fd2af75118815f after failing to start and attach to it"
2020-07-18T17:10:06.281 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 18 17:10:06 smithi070 bash[9216]: Error: container_linux.go:345: starting container process caused "exec: \"/usr/bin/ceph-mon\": stat /usr/bin/ceph-mon: no such file or directory"
2020-07-18T17:10:06.282 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 18 17:10:06 smithi070 bash[9216]: : OCI runtime error
2020-07-18T17:10:06.302 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 18 17:10:06 smithi070 systemd[1]: ceph-f930da78-c918-11ea-a06f-001a4aab830c@mon.b.service: main process exited, code=exited, status=127/n/a
2020-07-18T17:10:06.533 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 18 17:10:06 smithi070 podman[9457]: Error: no container with name or ID ceph-f930da78-c918-11ea-a06f-001a4aab830c-mon.b found: no such container
2020-07-18T17:10:06.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 18 17:10:06 smithi070 systemd[1]: Unit ceph-f930da78-c918-11ea-a06f-001a4aab830c@mon.b.service entered failed state.
2020-07-18T17:10:06.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 18 17:10:06 smithi070 systemd[1]: ceph-f930da78-c918-11ea-a06f-001a4aab830c@mon.b.service failed.
...
2020-07-18T17:10:10.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap...
</pre>
/a/kchai-2020-07-18_13:35:09-rados-wip-kefu-testing-2020-07-18-1927-distro-basic-smithi/5237560 Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1710922020-07-19T05:44:40ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>New</i></li></ul> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1711342020-07-20T09:50:54ZSebastian Wagner
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Resolved</i></li></ul><p>/a/kchai-2020-07-18_13:35:09-rados-wip-kefu-testing-2020-07-18-1927-distro-basic-smithi/5237560 is actually <a class="issue tracker-1 status-3 priority-6 priority-high2 closed" title="Bug: cephadm: error removing storage for container "...-mon": remove /var/lib/containers/storage/overl... (Resolved)" href="https://tracker.ceph.com/issues/46529">#46529</a></p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1776882020-10-22T11:42:40ZDeepika Upadhyay
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>New</i></li></ul><p>seems like this issue still exists as i see the fix PR was backported to octopus, seen this in:</p>
<p>/a/yuriw-2020-10-20_15:30:01-rados-wip-yuri5-testing-2020-10-07-1021-octopus-distro-basic-smithi/5542357/teuthology.log</p>
<pre>
msg="unable to remove container c7e4fd03b994ae13ba91fefb5d6955680ef4d535583dc08a5416267dd7c76a9c after failing to start and attach to it"
r_linux.go:345: starting container process caused "exec: \"/usr/bin/ceph-mon\": stat /usr/bin/ceph-mon: no such file or directory"
</pre> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1828062021-01-16T00:54:55ZDan Mickdmick@redhat.com
<ul></ul><p>I've been staring at that last failure for a few days, and I can't figure out what could possibly have caused it. The same image worked fine on smithi173, but on smithi157 it reports "/usr/bin/ceph-mon not found", and no apparent errors pulling. In fact I can't see where it was pulled at all from the log.</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1828072021-01-16T01:02:04ZSebastian Wagner
<ul></ul><p>it has nothing to do with the image. The issue went away with <a class="external" href="https://github.com/ceph/ceph/pull/35524">https://github.com/ceph/ceph/pull/35524</a> indicating something must have gone wrong cleaning up the previous container with the same name.</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1828632021-01-18T11:13:18ZSebastian Wagner
<ul><li><strong>Priority</strong> changed from <i>Urgent</i> to <i>Normal</i></li></ul><p>This is no longer critical, as it only affects octopus at this point.</p> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=1872712021-03-10T15:28:30ZSebastian Wagner
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Can't reproduce</i></li></ul> Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryhttps://tracker.ceph.com/issues/44990?journal_id=2058342021-11-11T11:23:53ZSebastian Wagner
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-1 priority-4 priority-default" href="/issues/53175">Bug #53175</a>: podman: failed to exec pid1: Exec format error: wrongly using the amd64-only digest</i> added</li></ul>