Project

General

Profile

Actions
Copy link

Bug #53424

closed

CEPHADM_DAEMON_PLACE_FAIL in orch:cephadm/mgr-nfs-upgrade/

Added by Sebastian Wagner over 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Sebastian Wagner
Category:
cephadm/nfs
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
44482
Crash signature (v1):
Crash signature (v2):

Description

mon[30568]: cephadm 2021-11-29T09:37:30.941127+0000 mgr.smithi198.ueaztz (mgr.24461) 46 : cephadm [INF] Removing key for client.nfs.foo.1.0.smithi198.mjthlo
mon[30568]: cephadm 2021-11-29T09:37:30.945462+0000 mgr.smithi198.ueaztz (mgr.24461) 47 : cephadm [INF] Removing key for client.nfs.foo.1.0.smithi198.mjthlo-rgw
mon[30568]: cephadm 2021-11-29T09:37:30.950752+0000 mgr.smithi198.ueaztz (mgr.24461) 48 : cephadm [ERR] Failed while placing nfs.foo.1.0.smithi198.mjthlo on smithi198: cephadm exited with an error code: 1, stde>
mon[30568]: /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
mon[30568]: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
mon[30568]: /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
mon[30568]: Deploy daemon nfs.foo.1.0.smithi198.mjthlo ...
mon[30568]: Verifying port 2049 ...
mon[30568]: Cannot bind to IP 0.0.0.0 port 2049: [Errno 98] Address already in use
mon[30568]: ERROR: TCP Port(s) '2049' required for nfs already in use
mon[30568]: cluster 2021-11-29T09:37:30.951587+0000 mgr.smithi198.ueaztz (mgr.24461) 49
mon[30568]:  : cluster [DBG] pgmap v22: 129 pgs: 129 active+clean; 316 MiB data, 950 MiB used, 706 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.7 MiB/s wr, 815 op/s
mon[30568]: cephadm 2021-11-29T09:37:30.953688+0000 mgr.smithi198.ueaztz (mgr.24461) 50 : cephadm [INF] Removing orphan daemon nfs.ganesha-foo.smithi112...
mon[30568]: cephadm 2021-11-29T09:37:30.953846+0000 mgr.smithi198.ueaztz (mgr.24461) 51 : cephadm [INF] Removing daemon nfs.ganesha-foo.smithi112 from smithi112
mon[30568]: cluster 2021-11-29
mon[30568]: T09:37:31.149845+0000 mon.smithi112 (mon.0) 877 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
mon

grep nfs.foo

➜  foo grep nfs.foo.1.0.smithi198 teuthology.log 
2021-11-29T09:37:30.295 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: cephadm 2021-11-29T09:37:29.528019+0000 mgr.smithi198.ueaztz (mgr.24461) 40 : cephadm [INF] Creating key for client.nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:30.295 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: audit 2021-11-29T09:37:29.528302+0000 mon.smithi198 (mon.1) 139 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch
2021-11-29T09:37:30.295 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: audit 2021-11-29T09:37:29.528611+0000 mon.smithi112 (mon.0) 865 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch
2021-11-29T09:37:30.295 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: audit 2021-11-29T09:37:29.531491+0000 mon.smithi112 (mon.0) 866 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished
2021-11-29T09:37:30.298 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: ) 43 : cephadm [INF] Creating key for client.nfs.foo.1.0.smithi198.mjthlo-rgw
2021-11-29T09:37:30.299 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: 143 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch
2021-11-29T09:37:30.299 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: 871 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch
2021-11-29T09:37:30.300 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: ) 872 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished
2021-11-29T09:37:30.302 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]:  (mgr.24461) 44 : cephadm [INF] Deploying daemon nfs.foo.1.0.smithi198.mjthlo on smithi198
2021-11-29T09:37:30.303 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:29 smithi112 conmon[30568]: [DBG] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "config get","who": "client.nfs.foo.1.0.smithi198.mjthlo","key": "container_image"}]: dispatch
2021-11-29T09:37:30.320 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: cephadm 2021-11-29T09:37:29.528019+0000 mgr.smithi198.ueaztz (mgr.24461) 40 : cephadm [INF] Creating key for client.nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:30.320 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: audit 2021-11-29T09:37:29.528302+0000 mon.smithi198 (mon.1) 139 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch
2021-11-29T09:37:30.320 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: audit 2021-11-29T09:37:29.528611+0000 mon.smithi112 (mon.0) 865 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch
2021-11-29T09:37:30.320 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: audit 2021-11-29T09:37:29.531491+0000 mon.smithi112 (mon.0) 866 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished
2021-11-29T09:37:30.324 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: [INF] Creating key for client.nfs.foo.1.0.smithi198.mjthlo-rgw
2021-11-29T09:37:30.325 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]:  from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch
2021-11-29T09:37:30.325 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: 11-29T09:37:29.654113+0000 mon.smithi112 (mon.0) 871 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch
2021-11-29T09:37:30.326 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: 0) 872 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished
2021-11-29T09:37:30.327 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]: 37:29.660464+0000 mgr.smithi198.ueaztz (mgr.24461) 44 : cephadm [INF] Deploying daemon nfs.foo.1.0.smithi198.mjthlo on smithi198
2021-11-29T09:37:30.328 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:29 smithi198 conmon[29563]:  from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "config get","who": "client.nfs.foo.1.0.smithi198.mjthlo","key": "container_image"}]: dispatch
2021-11-29T09:37:31.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]: audit 2021-11-29T09:37:30.941391+0000 mon.smithi198 (mon.1) 146 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo"}]: dispatch
2021-11-29T09:37:31.289 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]: audit 2021-11-29T09:37:30.941700+0000 mon.smithi112 (mon.0) 873 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo"}]: dispatch
2021-11-29T09:37:31.289 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]:  : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo"}]': finished
2021-11-29T09:37:31.290 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]: audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw"}]: dispatch
2021-11-29T09:37:31.290 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]: .0) 875 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw"}]: dispatch
2021-11-29T09:37:31.291 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]:  [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw"}]': finished
2021-11-29T09:37:31.564 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]: audit 2021-11-29T09:37:30.941391+0000 mon.smithi198 (mon.1) 146 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo"}]: dispatch
2021-11-29T09:37:31.565 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]: 873 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo"}]: dispatch
2021-11-29T09:37:31.565 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]:  (mon.0) 874 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo"}]': finished
2021-11-29T09:37:31.566 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]:  from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw"}]: dispatch
2021-11-29T09:37:31.567 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]: .0) 875 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw"}]: dispatch
2021-11-29T09:37:31.569 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]:  : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.foo.1.0.smithi198.mjthlo-rgw"}]': finished
2021-11-29T09:37:32.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: cephadm 2021-11-29T09:37:30.941127+0000 mgr.smithi198.ueaztz (mgr.24461) 46 : cephadm [INF] Removing key for client.nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:32.537 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: cephadm 2021-11-29T09:37:30.945462+0000 mgr.smithi198.ueaztz (mgr.24461) 47 : cephadm [INF] Removing key for client.nfs.foo.1.0.smithi198.mjthlo-rgw
2021-11-29T09:37:32.537 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: cephadm 2021-11-29T09:37:30.950752+0000 mgr.smithi198.ueaztz (mgr.24461) 48 : cephadm [ERR] Failed while placing nfs.foo.1.0.smithi198.mjthlo on smithi198: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:37:32.537 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:37:32.538 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:32.538 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:32.539 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: Deploy daemon nfs.foo.1.0.smithi198.mjthlo ...
2021-11-29T09:37:32.565 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: 24461) 46 : cephadm [INF] Removing key for client.nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:32.565 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: ) 47 : cephadm [INF] Removing key for client.nfs.foo.1.0.smithi198.mjthlo-rgw
2021-11-29T09:37:32.570 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: .24461) 48 : cephadm [ERR] Failed while placing nfs.foo.1.0.smithi198.mjthlo on smithi198: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:37:32.570 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:37:32.570 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:32.571 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:37:32.571 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: Deploy daemon nfs.foo.1.0.smithi198.mjthlo ...
2021-11-29T09:40:00.792 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:40:00 smithi112 conmon[30568]:  mon.smithi112 (mon.0) 933 : cluster [WRN]     Failed while placing nfs.foo.1.0.smithi198.mjthlo on smithi198: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:40:00.792 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:40:00 smithi112 conmon[30568]: 000295+0000 mon.smithi112 (mon.0) 934 : cluster [WRN] /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:40:00.792 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:40:00 smithi112 conmon[30568]: :40:00.000308+0000 mon.smithi112 (mon.0) 935 : cluster [WRN] Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:40:00.793 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:40:00 smithi112 conmon[30568]: mon.0) 936 : cluster [WRN] /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:40:00.793 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:40:00 smithi112 conmon[30568]: mon.smithi112 (mon.0) 937 : cluster [WRN] Deploy daemon nfs.foo.1.0.smithi198.mjthlo ...
2021-11-29T09:40:00.819 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:40:00 smithi198 conmon[29563]: mon.0) 933 : cluster [WRN]     Failed while placing nfs.foo.1.0.smithi198.mjthlo on smithi198: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:40:00.819 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:40:00 smithi198 conmon[29563]: 11-29T09:40:00.000295+0000 mon.smithi112 (mon.0) 934 : cluster [WRN] /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs-foo-1-0-smithi198-mjthlo
2021-11-29T09:40:00.820 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:40:00 smithi198 conmon[29563]: mon.0) 935 : cluster [WRN] Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:40:00.820 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:40:00 smithi198 conmon[29563]: 936 : cluster [WRN] /bin/podman: stderr Error: error inspecting object: no such container ceph-e6122430-50f6-11ec-8c2d-001a4aab830c-nfs.foo.1.0.smithi198.mjthlo
2021-11-29T09:40:00.820 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:40:00 smithi198 conmon[29563]: 00.000333+0000 mon.smithi112 (mon.0) 937 : cluster [WRN] Deploy daemon nfs.foo.1.0.smithi198.mjthlo ...

grep nfs.ganesha-foo.smithi112

➜  foo grep nfs.ganesha-foo.smithi112 teuthology.log 
2021-11-29T09:34:57.422 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:57 smithi112 conmon[30568]: .272588+0000 mon.smithi112 (mon.0) 684 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112", "caps": ["mon", "allow r", "osd", "allow rw pool=nfs-ganesha namespace=foo"]}]: dispatch
2021-11-29T09:34:57.422 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:57 smithi112 conmon[30568]:  (mon.0) 685 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112", "caps": ["mon", "allow r", "osd", "allow rw pool=nfs-ganesha namespace=foo"]}]': finished
2021-11-29T09:34:57.423 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:57 smithi112 conmon[30568]: 686 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch
2021-11-29T09:34:57.424 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:57 smithi112 conmon[30568]: ) 687 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished
2021-11-29T09:34:57.426 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:57 smithi112 conmon[30568]: 689 : audit [DBG] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd=[{"prefix": "config get", "who": "client.nfs.ganesha-foo.smithi112", "key": "container_image"}]: dispatch
2021-11-29T09:34:57.566 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:57 smithi198 conmon[29563]: audit 2021-11-29T09:34:57.272588+0000 mon.smithi112 (mon.0) 684 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112", "caps": ["mon", "allow r", "osd", "allow rw pool=nfs-ganesha namespace=foo"]}]: dispatch
2021-11-29T09:34:57.566 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:57 smithi198 conmon[29563]: ) 685 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112", "caps": ["mon", "allow r", "osd", "allow rw pool=nfs-ganesha namespace=foo"]}]': finished
2021-11-29T09:34:57.567 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:57 smithi198 conmon[29563]: audit 2021-11-29T09:34:57.276767+0000 mon.smithi112 (mon.0) 686 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch
2021-11-29T09:34:57.567 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:57 smithi198 conmon[29563]: audit 2021-11-29T09:34:57.279583+0000 mon.smithi112 (mon.0) 687 : audit [INF] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.ganesha-foo.smithi112-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished
2021-11-29T09:34:57.568 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:57 smithi198 conmon[29563]:  : audit [DBG] from='mgr.14164 172.21.15.112:0/3292037400' entity='mgr.smithi112.lrlmfr' cmd=[{"prefix": "config get", "who": "client.nfs.ganesha-foo.smithi112", "key": "container_image"}]: dispatch
2021-11-29T09:34:58.779 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:58 smithi112 conmon[30568]: cephadm 2021-11-29T09:34:57.272382+0000 mgr.smithi112.lrlmfr (mgr.14164) 239 : cephadm [INF] Create keyring: client.nfs.ganesha-foo.smithi112
2021-11-29T09:34:58.780 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:58 smithi112 conmon[30568]: cephadm 2021-11-29T09:34:57.276586+0000 mgr.smithi112.lrlmfr (mgr.14164) 241 : cephadm [INF] Create keyring: client.nfs.ganesha-foo.smithi112-rgw
2021-11-29T09:34:58.780 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:34:58 smithi112 conmon[30568]: cephadm 2021-11-29T09:34:57.280963+0000 mgr.smithi112.lrlmfr (mgr.14164) 242 : cephadm [INF] Deploying daemon nfs.ganesha-foo.smithi112 on smithi112
2021-11-29T09:34:58.814 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:58 smithi198 conmon[29563]: cephadm 2021-11-29T09:34:57.272382+0000 mgr.smithi112.lrlmfr (mgr.14164) 239 : cephadm [INF] Create keyring: client.nfs.ganesha-foo.smithi112
2021-11-29T09:34:58.814 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:58 smithi198 conmon[29563]: cephadm 2021-11-29T09:34:57.276586+0000 mgr.smithi112.lrlmfr (mgr.14164) 241 : cephadm [INF] Create keyring: client.nfs.ganesha-foo.smithi112-rgw
2021-11-29T09:34:58.815 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:34:58 smithi198 conmon[29563]: cephadm 2021-11-29T09:34:57.280963+0000 mgr.smithi112.lrlmfr (mgr.14164) 242 : cephadm [INF] Deploying daemon nfs.ganesha-foo.smithi112 on smithi112
2021-11-29T09:35:31.042 INFO:teuthology.orchestra.run.smithi112.stdout:nfs.ganesha-foo.smithi112   smithi112  running (30s)   24s ago    29s   3.3      docker.io/ceph/ceph:v15               2cf504fded39  5429025aa7a1
2021-11-29T09:36:02.393 INFO:teuthology.orchestra.run.smithi112.stdout:nfs.ganesha-foo.smithi112   smithi112  running (61s)  28s ago    61s  3.3      docker.io/ceph/ceph:v15               2cf504fded39  5429025aa7a1
2021-11-29T09:36:33.472 INFO:teuthology.orchestra.run.smithi112.stdout:nfs.ganesha-foo.smithi112   smithi112  running (92s)   59s ago    92s   3.3      docker.io/ceph/ceph:v15               2cf504fded39  5429025aa7a1
2021-11-29T09:37:05.391 INFO:teuthology.orchestra.run.smithi112.stdout:nfs.ganesha-foo.smithi112   smithi112         running (2m)     91s ago   2m        -        -  3.3                    2cf504fded39  5429025aa7a1
2021-11-29T09:37:31.292 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:31 smithi112 conmon[30568]:  mon.smithi198 (mon.1) 148 : audit [DBG] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "config get","who": "client.nfs.ganesha-foo.smithi112","key": "container_image"}]: dispatch
2021-11-29T09:37:31.569 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:31 smithi198 conmon[29563]:  [DBG] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "config get","who": "client.nfs.ganesha-foo.smithi112","key": "container_image"}]: dispatch
2021-11-29T09:37:32.541 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: cephadm 2021-11-29T09:37:30.953688+0000 mgr.smithi198.ueaztz (mgr.24461) 50 : cephadm [INF] Removing orphan daemon nfs.ganesha-foo.smithi112...
2021-11-29T09:37:32.542 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:32 smithi112 conmon[30568]: cephadm 2021-11-29T09:37:30.953846+0000 mgr.smithi198.ueaztz (mgr.24461) 51 : cephadm [INF] Removing daemon nfs.ganesha-foo.smithi112 from smithi112
2021-11-29T09:37:32.573 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]:  : cephadm [INF] Removing orphan daemon nfs.ganesha-foo.smithi112...
2021-11-29T09:37:32.574 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:32 smithi198 conmon[29563]: [INF] Removing daemon nfs.ganesha-foo.smithi112 from smithi112
2021-11-29T09:37:36.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:36 smithi112 conmon[30568]: audit 2021-11-29T09:37:36.245459+0000 mon.smithi198 (mon.1) 149 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112"}]: dispatch
2021-11-29T09:37:36.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:36 smithi112 conmon[30568]: audit 2021-11-29T09:37:36.245865+0000 mon.smithi112 (mon.0) 880 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112"}]: dispatch
2021-11-29T09:37:36.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:36 smithi112 conmon[30568]: audit 2021-11-29T09:37:36.250835+0000 mon.smithi112 (mon.0) 881 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112"}]': finished
2021-11-29T09:37:36.787 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:36 smithi112 conmon[30568]: audit 2021-11-29T09:37:36.252011+0000 mon.smithi198 (mon.1) 150 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112-rgw"}]: dispatch
2021-11-29T09:37:36.789 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:36 smithi112 conmon[30568]: audit 2021-11-29T09:37:36.252255+0000 mon.smithi112 (mon.0) 882 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112-rgw"}]: dispatch
2021-11-29T09:37:36.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:36 smithi112 conmon[30568]: audit 2021-11-29T09:37:36.255814+0000 mon.smithi112 (mon.0) 883 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112-rgw"}]': finished
2021-11-29T09:37:36.812 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:36 smithi198 conmon[29563]: 37:36.245459+0000 mon.smithi198 (mon.1) 149 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112"}]: dispatch
2021-11-29T09:37:36.812 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:36 smithi198 conmon[29563]: audit 2021-11-29T09:37:36.245865+0000 mon.smithi112 (mon.0) 880 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112"}]: dispatch
2021-11-29T09:37:36.812 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:36 smithi198 conmon[29563]: audit 2021-11-29T09:37:36.250835+0000 mon.smithi112 (mon.0) 881 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112"}]': finished
2021-11-29T09:37:36.812 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:36 smithi198 conmon[29563]: audit 2021-11-29T09:37:36.252011+0000 mon.smithi198 (mon.1) 150 : audit [INF] from='mgr.24461 172.21.15.198:0/1063122292' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112-rgw"}]: dispatch
2021-11-29T09:37:36.813 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:36 smithi198 conmon[29563]: audit 2021-11-29T09:37:36.252255+0000 mon.smithi112 (mon.0) 882 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd=[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112-rgw"}]: dispatch
2021-11-29T09:37:36.813 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:36 smithi198 conmon[29563]: audit 2021-11-29T09:37:36.255814+0000 mon.smithi112 (mon.0) 883 : audit [INF] from='mgr.24461 ' entity='mgr.smithi198.ueaztz' cmd='[{"prefix": "auth rm", "entity": "client.nfs.ganesha-foo.smithi112-rgw"}]': finished
2021-11-29T09:37:37.667 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:37 smithi198 conmon[29563]: cephadm 2021-11-29T09:37:36.245092+0000 mgr.smithi198.ueaztz (mgr.24461) 55 : cephadm [INF] Removing key for client.nfs.ganesha-foo.smithi112
2021-11-29T09:37:37.668 INFO:journalctl@ceph.mon.smithi198.smithi198.stdout:Nov 29 09:37:37 smithi198 conmon[29563]: cephadm 2021-11-29T09:37:36.251653+0000 mgr.smithi198.ueaztz (mgr.24461) 56 : cephadm [INF] Removing key for client.nfs.ganesha-foo.smithi112-rgw
2021-11-29T09:37:38.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:37 smithi112 conmon[30568]: [INF] Removing key for client.nfs.ganesha-foo.smithi112
2021-11-29T09:37:38.039 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Nov 29 09:37:37 smithi112 conmon[30568]: .24461) 56 : cephadm [INF] Removing key for client.nfs.ganesha-foo.smithi112-rgw

https://pulpito.ceph.com/swagner-2021-11-29_08:27:28-orch:cephadm-wip-swagner-testing-2021-11-26-1656-distro-default-smithi/6533368/

Related issues 2 (0 open — 2 closed)

Has duplicate Orchestrator - Bug #53545: rados/cephadm/mgr-nfs-upgrade failures due to CEPHADM_DAEMON_PLACE_FAILDuplicate

Actions
Copied to Orchestrator - Backport #55961: pacific: CEPHADM_DAEMON_PLACE_FAIL in orch:cephadm/mgr-nfs-upgrade/ResolvedActions
Actions
Copy link
 #2

Updated by Sebastian Wagner over 2 years ago

  • Has duplicate Bug #53545: rados/cephadm/mgr-nfs-upgrade failures due to CEPHADM_DAEMON_PLACE_FAIL added
Actions
Copy link
 #3

Updated by Laura Flores over 2 years ago

/a/yuriw-2021-12-22_22:11:35-rados-wip-yuri3-testing-2021-12-22-1047-distro-default-smithi/6580078

Actions
Copy link
 #4

Updated by Laura Flores over 2 years ago

/a/yuriw-2021-12-22_22:11:35-rados-wip-yuri3-testing-2021-12-22-1047-distro-default-smithi/6580296

Actions
Copy link
 #5

Updated by Laura Flores over 2 years ago

/a/yuriw-2021-12-23_16:50:03-rados-wip-yuri6-testing-2021-12-22-1410-distro-default-smithi/6582492

Actions
Copy link
 #6

Updated by Laura Flores over 2 years ago

/a/yuriw-2022-01-04_21:52:15-rados-wip-yuri7-testing-2022-01-04-1159-distro-default-smithi/6595248

Actions
Copy link
 #7

Updated by Sebastian Wagner over 2 years ago

And indeed we're not stopping or undeploying the old ganesha:

2022-01-04T23:12:06.766 INFO:journalctl@ceph.mon.smithi109.smithi109.stdout:Jan 04 23:12:06 smithi109 conmon[28389]:  [INF] Deploying daemon nfs.ganesha-foo.smithi155 on smithi155
2022-01-04 23:12:05,743 DEBUG --------------------------------------------------------------------------------
cephadm ['--image', 'docker.io/ceph/ceph:v15', '--no-container-init', 'deploy', '--fsid', '056557d4-6db3-11ec-8c32-001a4aab830c', '--name', 'nfs.ganesha-foo.smithi155', '--allow-ptrace', '--config-json', '-']
2022-01-04 23:12:05,750 DEBUG container_init=False
2022-01-04 23:12:05,750 DEBUG Acquiring lock 140458404686984 on /run/cephadm/056557d4-6db3-11ec-8c32-001a4aab830c.lock
2022-01-04 23:12:05,750 DEBUG Lock 140458404686984 acquired on /run/cephadm/056557d4-6db3-11ec-8c32-001a4aab830c.lock
2022-01-04 23:12:05,750 DEBUG Running command: systemctl is-enabled ceph-056557d4-6db3-11ec-8c32-001a4aab830c@nfs.ganesha-foo.smithi155
2022-01-04 23:12:05,758 DEBUG systemctl: stdout disabled
2022-01-04 23:12:05,758 DEBUG Running command: systemctl is-active ceph-056557d4-6db3-11ec-8c32-001a4aab830c@nfs.ganesha-foo.smithi155
2022-01-04 23:12:05,768 DEBUG systemctl: stdout inactive
2022-01-04 23:12:05,768 DEBUG Running command: /bin/podman ps --format {{.Names}}
2022-01-04 23:12:05,821 DEBUG /bin/podman: stdout ceph-056557d4-6db3-11ec-8c32-001a4aab830c-mon.smithi155
2022-01-04 23:12:05,822 DEBUG /bin/podman: stdout ceph-056557d4-6db3-11ec-8c32-001a4aab830c-mgr.smithi155.dyqjgc
2022-01-04 23:12:05,822 DEBUG /bin/podman: stdout ceph-056557d4-6db3-11ec-8c32-001a4aab830c-node-exporter.smithi155
2022-01-04 23:12:05,822 DEBUG /bin/podman: stdout ceph-056557d4-6db3-11ec-8c32-001a4aab830c-prometheus.smithi155
2022-01-04 23:12:05,822 DEBUG /bin/podman: stdout ceph-056557d4-6db3-11ec-8c32-001a4aab830c-crash.smithi155
2022-01-04 23:12:05,822 DEBUG /bin/podman: stdout ceph-056557d4-6db3-11ec-8c32-001a4aab830c-osd.4
2022-01-04 23:12:05,822 DEBUG /bin/podman: stdout ceph-056557d4-6db3-11ec-8c32-001a4aab830c-osd.5
2022-01-04 23:12:05,822 DEBUG /bin/podman: stdout ceph-056557d4-6db3-11ec-8c32-001a4aab830c-osd.6
2022-01-04 23:12:05,822 DEBUG /bin/podman: stdout ceph-056557d4-6db3-11ec-8c32-001a4aab830c-osd.7
2022-01-04 23:12:05,822 DEBUG /bin/podman: stdout ceph-056557d4-6db3-11ec-8c32-001a4aab830c-mds.foofs.smithi155.rtvdwp
2022-01-04 23:12:05,898 INFO Deploy daemon nfs.ganesha-foo.smithi155 ...
2022-01-04 23:12:05,898 DEBUG Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint stat -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=smithi155 docker.io/ceph/ceph:v15 -c %u %g /var/lib/ceph
2022-01-04 23:12:06,343 DEBUG stat: stdout 167 167
2022-01-04 23:12:06,832 INFO Verifying port 2049 ...
2022-01-04 23:12:06,833 INFO Creating ganesha config...
2022-01-04 23:12:06,833 INFO Write file: /var/lib/ceph/056557d4-6db3-11ec-8c32-001a4aab830c/nfs.ganesha-foo.smithi155/etc/ganesha/ganesha.conf
2022-01-04 23:12:06,833 INFO Creating RADOS grace for action: add
2022-01-04 23:12:06,834 INFO Creating RADOS grace for action: remove
2022-01-04 23:12:06,834 DEBUG Running command: systemctl daemon-reload
2022-01-04 23:12:06,937 DEBUG Running command: systemctl stop ceph-056557d4-6db3-11ec-8c32-001a4aab830c@nfs.ganesha-foo.smithi155
2022-01-04 23:12:06,964 DEBUG Running command: systemctl reset-failed ceph-056557d4-6db3-11ec-8c32-001a4aab830c@nfs.ganesha-foo.smithi155
2022-01-04 23:12:06,969 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-056557d4-6db3-11ec-8c32-001a4aab830c@nfs.ganesha-foo.smithi155.service: Unit ceph-056557d4-6db3-11ec-8c32-001a4aab830c@nfs.ganesha-foo.smithi155.service not loaded.
2022-01-04 23:12:06,969 DEBUG Running command: systemctl enable ceph-056557d4-6db3-11ec-8c32-001a4aab830c@nfs.ganesha-foo.smithi155
2022-01-04 23:12:06,975 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-056557d4-6db3-11ec-8c32-001a4aab830c.target.wants/ceph-056557d4-6db3-11ec-8c32-001a4aab830c@nfs.ganesha-foo.smithi155.service → /etc/systemd/system/ceph-056557d4-6db3-11ec-8c32-001a4aab830c@.service.
2022-01-04 23:12:07,067 DEBUG Running command: systemctl start ceph-056557d4-6db3-11ec-8c32-001a4aab830c@nfs.ganesha-foo.smithi155
2022-01-04 23:12:08,500 DEBUG Running command: systemctl is-enabled firewalld.service
2022-01-04 23:12:08,508 DEBUG systemctl: stdout disabled
2022-01-04 23:12:08,508 DEBUG Running command: systemctl is-active firewalld.service
2022-01-04 23:12:08,514 DEBUG systemctl: stdout inactive
2022-01-04 23:12:08,514 DEBUG firewalld.service is not enabled
2022-01-04 23:12:08,514 DEBUG Not possible to enable service <nfs>. firewalld.service is not available
2022-01-04 23:12:08,514 DEBUG Not possible to open ports <[]>. firewalld.service is not available
2022-01-04 23:12:08,514 DEBUG Running command: systemctl is-enabled firewalld.service
2022-01-04 23:12:08,521 DEBUG systemctl: stdout disabled
2022-01-04 23:12:08,521 DEBUG Running command: systemctl is-active firewalld.service
2022-01-04 23:12:08,526 DEBUG systemctl: stdout inactive
2022-01-04 23:12:08,527 DEBUG firewalld.service is not enabled
2022-01-04 23:12:08,527 DEBUG Not possible to open ports <[2049]>. firewalld.service is not available
2022-01-04 23:12:08,527 DEBUG Releasing lock 140458404686984 on /run/cephadm/056557d4-6db3-11ec-8c32-001a4aab830c.lock
2022-01-04 23:12:08,527 DEBUG Lock 140458404686984 released on /run/cephadm/056557d4-6db3-11ec-8c32-001a4aab830c.lock
...snip...
2022-01-04 23:14:34,373 7f0fde4e5b80 DEBUG --------------------------------------------------------------------------------
cephadm ['--image', 'docker.io/ceph/ceph@sha256:0368cf225b3a13b7bdeb3d81ecf370a62931ffa5ff87af880d66aebae74f910a', '--no-container-init', 'deploy', '--fsid', '056557d4-6db3-11ec-8c32-001a4aab830c', '--name', 'nfs.foo.1.0.smithi155.waoktu', '--meta-json', '{"service_name": "nfs.foo", "ports": [], "ip": null, "deployed_by": ["quay.ceph.io/ceph-ci/ceph@sha256:b263c1f8b7c4f1dbd1a1105a1a06fac40bcf207870d4a077aaa1f7c650920968"], "rank": 1, "rank_generation": 0}', '--config-json', '-', '--allow-ptrace']
2022-01-04 23:14:34,425 7f0fde4e5b80 DEBUG /bin/podman: 3.4.1-dev
2022-01-04 23:14:34,518 7f0fde4e5b80 DEBUG Acquiring lock 139706129363072 on /run/cephadm/056557d4-6db3-11ec-8c32-001a4aab830c.lock
2022-01-04 23:14:34,518 7f0fde4e5b80 DEBUG Lock 139706129363072 acquired on /run/cephadm/056557d4-6db3-11ec-8c32-001a4aab830c.lock
2022-01-04 23:14:34,534 7f0fde4e5b80 DEBUG systemctl: disabled
2022-01-04 23:14:34,548 7f0fde4e5b80 DEBUG systemctl: inactive
2022-01-04 23:14:34,601 7f0fde4e5b80 DEBUG /bin/podman: Error: error inspecting object: no such container ceph-056557d4-6db3-11ec-8c32-001a4aab830c-nfs-foo-1-0-smithi155-waoktu
2022-01-04 23:14:34,605 7f0fde4e5b80 INFO Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-056557d4-6db3-11ec-8c32-001a4aab830c-nfs-foo-1-0-smithi155-waoktu
2022-01-04 23:14:34,605 7f0fde4e5b80 INFO /bin/podman: stderr Error: error inspecting object: no such container ceph-056557d4-6db3-11ec-8c32-001a4aab830c-nfs-foo-1-0-smithi155-waoktu
2022-01-04 23:14:34,651 7f0fde4e5b80 DEBUG /bin/podman: Error: error inspecting object: no such container ceph-056557d4-6db3-11ec-8c32-001a4aab830c-nfs.foo.1.0.smithi155.waoktu
2022-01-04 23:14:34,653 7f0fde4e5b80 INFO Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-056557d4-6db3-11ec-8c32-001a4aab830c-nfs.foo.1.0.smithi155.waoktu
2022-01-04 23:14:34,653 7f0fde4e5b80 INFO /bin/podman: stderr Error: error inspecting object: no such container ceph-056557d4-6db3-11ec-8c32-001a4aab830c-nfs.foo.1.0.smithi155.waoktu
2022-01-04 23:14:34,653 7f0fde4e5b80 INFO Deploy daemon nfs.foo.1.0.smithi155.waoktu ...
2022-01-04 23:14:35,079 7f0fde4e5b80 DEBUG stat: 167 167
2022-01-04 23:14:35,394 7f0fde4e5b80 INFO Verifying port 2049 ...
2022-01-04 23:14:35,395 7f0fde4e5b80 WARNING Cannot bind to IP 0.0.0.0 port 2049: [Errno 98] Address already in use
2022-01-04 23:14:35,395 7f0fde4e5b80 ERROR ERROR: TCP Port(s) '2049' required for nfs already in use
----...snip....
2022-01-04T23:14:42.862 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Jan 04 23:14:42 smithi155 conmon[27505]: [INF] Removing daemon nfs.ganesha-foo.smithi155 from smithi155

And indeed we're not removing the old nfs daemons:

➜  foo grep 'Removing old nfs' teuthology.log 
➜  foo 
➜  foo grep 'Replacing nfs.ganesha' teuthology.log
2022-01-04T23:14:33.440 INFO:journalctl@ceph.mon.smithi109.smithi109.stdout:Jan 04 23:14:33 smithi109 conmon[28389]: cephadm 2022-01-04T23:14:32.236386+0000 mgr.smithi155.dyqjgc (mgr.14616) 22 : cephadm [INF] Replacing nfs.ganesha-foo with nfs.foo
Actions
Copy link
 #8

Updated by Sebastian Wagner over 2 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Sebastian Wagner
  • Pull request ID set to 44482
Actions
Copy link
 #9

Updated by Laura Flores over 2 years ago

/a/yuriw-2022-01-06_15:50:38-rados-wip-yuri8-testing-2022-01-05-1411-distro-default-smithi/6598788

Actions
Copy link
 #10

Updated by Laura Flores over 2 years ago

/a/yuriw-2022-01-11_19:17:55-rados-wip-yuri5-testing-2022-01-11-0843-distro-default-smithi/6608761

Actions
Copy link
 #11

Updated by Aishwarya Mathuria over 2 years ago

/a/yuriw-2022-01-13_18:06:52-rados-wip-yuri3-testing-2022-01-13-0809-distro-default-smithi/6614482

Actions
Copy link
 #12

Updated by Sebastian Wagner over 2 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #13

Updated by Redouane Kachach Elhichou almost 2 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link
 #14

Updated by Redouane Kachach Elhichou almost 2 years ago

  • Status changed from Resolved to Pending Backport
  • Backport set to pacific
Actions
Copy link
 #15

Updated by Backport Bot almost 2 years ago

  • Copied to Backport #55961: pacific: CEPHADM_DAEMON_PLACE_FAIL in orch:cephadm/mgr-nfs-upgrade/ added
Actions
Copy link
 #16

Updated by Adam King almost 2 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF