Actions
Bug #53424
closedCEPHADM_DAEMON_PLACE_FAIL in orch:cephadm/mgr-nfs-upgrade/
% Done:
0%
Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Description
mon[30568]: cephadm 2021-11-29T09:37:30.941127+0000 mgr.smithi198.ueaztz (mgr.24461) 46 : cephadm [INF] Removing key for client.nfs.foo.1.0.smithi198.mjthlo
mon[30568]: cephadm 2021-11-29T09:37:30.945462+0000 mgr.smithi198.ueaztz (mgr.24461) 47 : cephadm [INF] Removing key for client.nfs.foo.1.0.smithi198.mjthlo-rgw
mon[30568]: cephadm 2021-11-29T09:37:30.950752+0000 mgr.smithi198.ueaztz (mgr.24461) 48 : cephadm [ERR] Failed while placing nfs.foo.1.0.smithi198.mjthlo on smithi198: cephadm exited with an error code: 1, stde>
mon[30568]: /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
mon[30568]: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
mon[30568]: /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
mon[30568]: Deploy daemon nfs.foo.1.0.smithi198.mjthlo ...
mon[30568]: Verifying port 2049 ...
mon[30568]: Cannot bind to IP 0.0.0.0 port 2049: [Errno 98] Address already in use
mon[30568]: ERROR: TCP Port(s) '2049' required for nfs already in use
mon[30568]: cluster 2021-11-29T09:37:30.951587+0000 mgr.smithi198.ueaztz (mgr.24461) 49
mon[30568]: : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 316 MiB data, 950 MiB used, 706 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.7 MiB/s wr, 815 op/s
mon[30568]: cephadm 2021-11-29T09:37:30.953688+0000 mgr.smithi198.ueaztz (mgr.24461) 50 : cephadm [INF] Removing orphan daemon nfs.ganesha-foo.smithi112...
mon[30568]: cephadm 2021-11-29T09:37:30.953846+0000 mgr.smithi198.ueaztz (mgr.24461) 51 : cephadm [INF] Removing daemon nfs.ganesha-foo.smithi112 from smithi112
mon[30568]: cluster 2021-11-29
mon[30568]: T09:37:31.149845+0000 mon.smithi112 (mon.0) 877 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
mon
grep nfs.foo
➜ foo grep nfs.foo.1.0.smithi198 teuthology.log
2021-11-29T09:37:30.295 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: cephadm 2021-11-29T09:37:29.528019+0000 mgr.smithi198.ueaztz (mgr.24461) 40 : cephadm [INF] Creating key for client.nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:30.295 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: audit 2021-11-29T09:37:29.528302+0000 mon.smithi198 (mon.1) 139 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch
2021-11-29T09:37:30.295 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: audit 2021-11-29T09:37:29.528611+0000 mon.smithi112 (mon.0) 865 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch
2021-11-29T09:37:30.295 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: audit 2021-11-29T09:37:29.531491+0000 mon.smithi112 (mon.0) 866 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished
2021-11-29T09:37:30.298 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: ) 43 : cephadm [INF] Creating key for client.nfs.foo.1.0.smithi198.mjthlo-rgw
2021-11-29T09:37:30.299 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: 143 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch
2021-11-29T09:37:30.299 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: 871 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch
2021-11-29T09:37:30.300 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: ) 872 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished
2021-11-29T09:37:30.302 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: (mgr.24461) 44 : cephadm [INF] Deploying daemon nfs.foo.1.0.smithi198.mjthlo on smithi198
2021-11-29T09:37:30.303 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: [DBG] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "config get","who": "client.nfs.foo.1.0.smithi198.mjthlo","key": "container_image"}]: dispatch
2021-11-29T09:37:30.320 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: cephadm 2021-11-29T09:37:29.528019+0000 mgr.smithi198.ueaztz (mgr.24461) 40 : cephadm [INF] Creating key for client.nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:30.320 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: audit 2021-11-29T09:37:29.528302+0000 mon.smithi198 (mon.1) 139 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch
2021-11-29T09:37:30.320 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: audit 2021-11-29T09:37:29.528611+0000 mon.smithi112 (mon.0) 865 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch
2021-11-29T09:37:30.320 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: audit 2021-11-29T09:37:29.531491+0000 mon.smithi112 (mon.0) 866 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished
2021-11-29T09:37:30.324 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: [INF] Creating key for client.nfs.foo.1.0.smithi198.mjthlo-rgw
2021-11-29T09:37:30.325 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch
2021-11-29T09:37:30.325 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: 11-29T09:37:29.654113+0000 mon.smithi112 (mon.0) 871 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch
2021-11-29T09:37:30.326 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: 0) 872 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished
2021-11-29T09:37:30.327 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: 37:29.660464+0000 mgr.smithi198.ueaztz (mgr.24461) 44 : cephadm [INF] Deploying daemon nfs.foo.1.0.smithi198.mjthlo on smithi198
2021-11-29T09:37:30.328 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "config get","who": "client.nfs.foo.1.0.smithi198.mjthlo","key": "container_image"}]: dispatch
2021-11-29T09:37:31.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]: audit 2021-11-29T09:37:30.941391+0000 mon.smithi198 (mon.1) 146 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo"}]: dispatch
2021-11-29T09:37:31.289 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]: audit 2021-11-29T09:37:30.941700+0000 mon.smithi112 (mon.0) 873 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo"}]: dispatch
2021-11-29T09:37:31.289 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]: : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo"}]': finished
2021-11-29T09:37:31.290 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]: audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw"}]: dispatch
2021-11-29T09:37:31.290 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]: .0) 875 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw"}]: dispatch
2021-11-29T09:37:31.291 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]: [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw"}]': finished
2021-11-29T09:37:31.564 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]: audit 2021-11-29T09:37:30.941391+0000 mon.smithi198 (mon.1) 146 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo"}]: dispatch
2021-11-29T09:37:31.565 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]: 873 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo"}]: dispatch
2021-11-29T09:37:31.565 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]: (mon.0) 874 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo"}]': finished
2021-11-29T09:37:31.566 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]: from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw"}]: dispatch
2021-11-29T09:37:31.567 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]: .0) 875 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw"}]: dispatch
2021-11-29T09:37:31.569 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]: : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw"}]': finished
2021-11-29T09:37:32.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: cephadm 2021-11-29T09:37:30.941127+0000 mgr.smithi198.ueaztz (mgr.24461) 46 : cephadm [INF] Removing key for client.nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:32.537 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: cephadm 2021-11-29T09:37:30.945462+0000 mgr.smithi198.ueaztz (mgr.24461) 47 : cephadm [INF] Removing key for client.nfs.foo.1.0.smithi198.mjthlo-rgw
2021-11-29T09:37:32.537 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: cephadm 2021-11-29T09:37:30.950752+0000 mgr.smithi198.ueaztz (mgr.24461) 48 : cephadm [ERR] Failed while placing nfs.foo.1.0.smithi198.mjthlo on smithi198: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:37:32.537 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:37:32.538 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:32.538 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:32.539 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: Deploy daemon nfs.foo.1.0.smithi198.mjthlo ...
2021-11-29T09:37:32.565 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: 24461) 46 : cephadm [INF] Removing key for client.nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:32.565 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: ) 47 : cephadm [INF] Removing key for client.nfs.foo.1.0.smithi198.mjthlo-rgw
2021-11-29T09:37:32.570 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: .24461) 48 : cephadm [ERR] Failed while placing nfs.foo.1.0.smithi198.mjthlo on smithi198: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:37:32.570 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:37:32.570 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:32.571 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:32.571 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: Deploy daemon nfs.foo.1.0.smithi198.mjthlo ...
2021-11-29T09:40:00.792 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:40:00 smithi112 conmon[30568]: mon.smithi112 (mon.0) 933 : cluster [WRN] Failed while placing nfs.foo.1.0.smithi198.mjthlo on smithi198: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:40:00.792 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:40:00 smithi112 conmon[30568]: 000295+0000 mon.smithi112 (mon.0) 934 : cluster [WRN] /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:40:00.792 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:40:00 smithi112 conmon[30568]: :40:00.000308+0000 mon.smithi112 (mon.0) 935 : cluster [WRN] Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:40:00.793 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:40:00 smithi112 conmon[30568]: mon.0) 936 : cluster [WRN] /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:40:00.793 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:40:00 smithi112 conmon[30568]: mon.smithi112 (mon.0) 937 : cluster [WRN] Deploy daemon nfs.foo.1.0.smithi198.mjthlo ...
2021-11-29T09:40:00.819 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:40:00 smithi198 conmon[29563]: mon.0) 933 : cluster [WRN] Failed while placing nfs.foo.1.0.smithi198.mjthlo on smithi198: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:40:00.819 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:40:00 smithi198 conmon[29563]: 11-29T09:40:00.000295+0000 mon.smithi112 (mon.0) 934 : cluster [WRN] /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:40:00.820 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:40:00 smithi198 conmon[29563]: mon.0) 935 : cluster [WRN] Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:40:00.820 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:40:00 smithi198 conmon[29563]: 936 : cluster [WRN] /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:40:00.820 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:40:00 smithi198 conmon[29563]: 00.000333+0000 mon.smithi112 (mon.0) 937 : cluster [WRN] Deploy daemon nfs.foo.1.0.smithi198.mjthlo ...
grep nfs.ganesha-foo.smithi112
➜ foo grep nfs.ganesha-foo.smithi112 teuthology.log
2021-11-29T09:34:57.422 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:57 smithi112 conmon[30568]: .272588+0000 mon.smithi112 (mon.0) 684 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112", "caps": ["mon", "allow r", "osd", "allow rw pool=nfs-ganesha namespace=foo"]}]: dispatch
2021-11-29T09:34:57.422 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:57 smithi112 conmon[30568]: (mon.0) 685 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112", "caps": ["mon", "allow r", "osd", "allow rw pool=nfs-ganesha namespace=foo"]}]': finished
2021-11-29T09:34:57.423 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:57 smithi112 conmon[30568]: 686 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch
2021-11-29T09:34:57.424 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:57 smithi112 conmon[30568]: ) 687 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished
2021-11-29T09:34:57.426 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:57 smithi112 conmon[30568]: 689 : audit [DBG] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd=[{"prefix": "config get", "who": "client.nfs.ganesha-foo.smithi112", "key": "container_image"}]: dispatch
2021-11-29T09:34:57.566 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:57 smithi198 conmon[29563]: audit 2021-11-29T09:34:57.272588+0000 mon.smithi112 (mon.0) 684 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112", "caps": ["mon", "allow r", "osd", "allow rw pool=nfs-ganesha namespace=foo"]}]: dispatch
2021-11-29T09:34:57.566 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:57 smithi198 conmon[29563]: ) 685 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112", "caps": ["mon", "allow r", "osd", "allow rw pool=nfs-ganesha namespace=foo"]}]': finished
2021-11-29T09:34:57.567 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:57 smithi198 conmon[29563]: audit 2021-11-29T09:34:57.276767+0000 mon.smithi112 (mon.0) 686 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch
2021-11-29T09:34:57.567 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:57 smithi198 conmon[29563]: audit 2021-11-29T09:34:57.279583+0000 mon.smithi112 (mon.0) 687 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished
2021-11-29T09:34:57.568 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:57 smithi198 conmon[29563]: : audit [DBG] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd=[{"prefix": "config get", "who": "client.nfs.ganesha-foo.smithi112", "key": "container_image"}]: dispatch
2021-11-29T09:34:58.779 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:58 smithi112 conmon[30568]: cephadm 2021-11-29T09:34:57.272382+0000 mgr.smithi112.lrlmfr (mgr.14164) 239 : cephadm [INF] Create keyring: client.nfs.ganesha-foo.smithi112
2021-11-29T09:34:58.780 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:58 smithi112 conmon[30568]: cephadm 2021-11-29T09:34:57.276586+0000 mgr.smithi112.lrlmfr (mgr.14164) 241 : cephadm [INF] Create keyring: client.nfs.ganesha-foo.smithi112-rgw
2021-11-29T09:34:58.780 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:58 smithi112 conmon[30568]: cephadm 2021-11-29T09:34:57.280963+0000 mgr.smithi112.lrlmfr (mgr.14164) 242 : cephadm [INF] Deploying daemon nfs.ganesha-foo.smithi112 on smithi112
2021-11-29T09:34:58.814 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:58 smithi198 conmon[29563]: cephadm 2021-11-29T09:34:57.272382+0000 mgr.smithi112.lrlmfr (mgr.14164) 239 : cephadm [INF] Create keyring: client.nfs.ganesha-foo.smithi112
2021-11-29T09:34:58.814 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:58 smithi198 conmon[29563]: cephadm 2021-11-29T09:34:57.276586+0000 mgr.smithi112.lrlmfr (mgr.14164) 241 : cephadm [INF] Create keyring: client.nfs.ganesha-foo.smithi112-rgw
2021-11-29T09:34:58.815 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:58 smithi198 conmon[29563]: cephadm 2021-11-29T09:34:57.280963+0000 mgr.smithi112.lrlmfr (mgr.14164) 242 : cephadm [INF] Deploying daemon nfs.ganesha-foo.smithi112 on smithi112
2021-11-29T09:35:31.042 INFO:teuthology.orchestra.run.smithi112.stdout:nfs.ganesha-foo.smithi112 smithi112 running (30s) 24s ago 29s 3.3 docker.io/ceph/ceph:v15 2cf504fded39 5429025aa7a1
2021-11-29T09:36:02.393 INFO:teuthology.orchestra.run.smithi112.stdout:nfs.ganesha-foo.smithi112 smithi112 running (61s) 28s ago 61s 3.3 docker.io/ceph/ceph:v15 2cf504fded39 5429025aa7a1
2021-11-29T09:36:33.472 INFO:teuthology.orchestra.run.smithi112.stdout:nfs.ganesha-foo.smithi112 smithi112 running (92s) 59s ago 92s 3.3 docker.io/ceph/ceph:v15 2cf504fded39 5429025aa7a1
2021-11-29T09:37:05.391 INFO:teuthology.orchestra.run.smithi112.stdout:nfs.ganesha-foo.smithi112 smithi112 running (2m) 91s ago 2m - - 3.3 2cf504fded39 5429025aa7a1
2021-11-29T09:37:31.292 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]: mon.smithi198 (mon.1) 148 : audit [DBG] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "config get","who": "client.nfs.ganesha-foo.smithi112","key": "container_image"}]: dispatch
2021-11-29T09:37:31.569 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]: [DBG] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "config get","who": "client.nfs.ganesha-foo.smithi112","key": "container_image"}]: dispatch
2021-11-29T09:37:32.541 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: cephadm 2021-11-29T09:37:30.953688+0000 mgr.smithi198.ueaztz (mgr.24461) 50 : cephadm [INF] Removing orphan daemon nfs.ganesha-foo.smithi112...
2021-11-29T09:37:32.542 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: cephadm 2021-11-29T09:37:30.953846+0000 mgr.smithi198.ueaztz (mgr.24461) 51 : cephadm [INF] Removing daemon nfs.ganesha-foo.smithi112 from smithi112
2021-11-29T09:37:32.573 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: : cephadm [INF] Removing orphan daemon nfs.ganesha-foo.smithi112...
2021-11-29T09:37:32.574 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: [INF] Removing daemon nfs.ganesha-foo.smithi112 from smithi112
2021-11-29T09:37:36.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:36 smithi112 conmon[30568]: audit 2021-11-29T09:37:36.245459+0000 mon.smithi198 (mon.1) 149 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112"}]: dispatch
2021-11-29T09:37:36.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:36 smithi112 conmon[30568]: audit 2021-11-29T09:37:36.245865+0000 mon.smithi112 (mon.0) 880 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112"}]: dispatch
2021-11-29T09:37:36.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:36 smithi112 conmon[30568]: audit 2021-11-29T09:37:36.250835+0000 mon.smithi112 (mon.0) 881 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112"}]': finished
2021-11-29T09:37:36.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:36 smithi112 conmon[30568]: audit 2021-11-29T09:37:36.252011+0000 mon.smithi198 (mon.1) 150 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112-rgw"}]: dispatch
2021-11-29T09:37:36.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:36 smithi112 conmon[30568]: audit 2021-11-29T09:37:36.252255+0000 mon.smithi112 (mon.0) 882 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112-rgw"}]: dispatch
2021-11-29T09:37:36.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:36 smithi112 conmon[30568]: audit 2021-11-29T09:37:36.255814+0000 mon.smithi112 (mon.0) 883 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112-rgw"}]': finished
2021-11-29T09:37:36.812 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:36 smithi198 conmon[29563]: 37:36.245459+0000 mon.smithi198 (mon.1) 149 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112"}]: dispatch
2021-11-29T09:37:36.812 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:36 smithi198 conmon[29563]: audit 2021-11-29T09:37:36.245865+0000 mon.smithi112 (mon.0) 880 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112"}]: dispatch
2021-11-29T09:37:36.812 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:36 smithi198 conmon[29563]: audit 2021-11-29T09:37:36.250835+0000 mon.smithi112 (mon.0) 881 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112"}]': finished
2021-11-29T09:37:36.812 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:36 smithi198 conmon[29563]: audit 2021-11-29T09:37:36.252011+0000 mon.smithi198 (mon.1) 150 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112-rgw"}]: dispatch
2021-11-29T09:37:36.813 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:36 smithi198 conmon[29563]: audit 2021-11-29T09:37:36.252255+0000 mon.smithi112 (mon.0) 882 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112-rgw"}]: dispatch
2021-11-29T09:37:36.813 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:36 smithi198 conmon[29563]: audit 2021-11-29T09:37:36.255814+0000 mon.smithi112 (mon.0) 883 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112-rgw"}]': finished
2021-11-29T09:37:37.667 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:37 smithi198 conmon[29563]: cephadm 2021-11-29T09:37:36.245092+0000 mgr.smithi198.ueaztz (mgr.24461) 55 : cephadm [INF] Removing key for client.nfs.ganesha-foo.smithi112
2021-11-29T09:37:37.668 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:37 smithi198 conmon[29563]: cephadm 2021-11-29T09:37:36.251653+0000 mgr.smithi198.ueaztz (mgr.24461) 56 : cephadm [INF] Removing key for client.nfs.ganesha-foo.smithi112-rgw
2021-11-29T09:37:38.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:37 smithi112 conmon[30568]: [INF] Removing key for client.nfs.ganesha-foo.smithi112
2021-11-29T09:37:38.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:37 smithi112 conmon[30568]: .24461) 56 : cephadm [INF] Removing key for client.nfs.ganesha-foo.smithi112-rgw
Actions