Ceph » Orchestrator

Configuring journald as the log driver allows for the conmon logs to show the actual output from the failed daemon container.

# cephadm logs --name nfs.bar.host1 -- -n 30 
Inferring fsid 9cfc6bda-89b5-413b-830f-01c8ddad63ec
-- Logs begin at Sat 2020-10-03 13:38:32 MDT, end at Tue 2020-10-20 11:07:23 MDT. --
Oct 14 08:22:14 host1 systemd[1]: Stopped Ceph nfs.bar.host1 for 9cfc6bda-89b5-413b-830f-01c8ddad63ec.
Oct 14 08:22:14 host1 systemd[1]: ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec@nfs.bar.host1.service: Start request repeated too quickly.
Oct 14 08:22:14 host1 systemd[1]: ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec@nfs.bar.host1.service: Failed with result 'exit-code'.
Oct 14 08:22:14 host1 systemd[1]: Failed to start Ceph nfs.bar.host1 for 9cfc6bda-89b5-413b-830f-01c8ddad63ec.
Oct 20 11:04:59 host1 systemd[1]: Starting Ceph nfs.bar.host1 for 9cfc6bda-89b5-413b-830f-01c8ddad63ec...
Oct 20 11:04:59 host1 podman[16752]: Error: no container with name or ID ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec-nfs.bar.host1 found: no such container
Oct 20 11:04:59 host1 bash[16798]: Error: Failed to evict container: "": Failed to find container "ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec-nfs.bar.host1-grace-add" in state: no container with name or ID ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec-nfs.bar.host1-grace-add found: no such container
Oct 20 11:04:59 host1 bash[16842]: Error: no container with ID or name "ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec-nfs.bar.host1-grace-add" found: no such container
Oct 20 11:05:38 host1 bash[17068]: Error: Failed to evict container: "": Failed to find container "ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec-nfs.bar.host1" in state: no container with name or ID ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec-nfs.bar.host1 found: no such container
Oct 20 11:05:38 host1 bash[17113]: Error: no container with ID or name "ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec-nfs.bar.host1" found: no such container
Oct 20 11:05:53 host1 bash[17158]: d72d1dd29626b9970460937cfbdf15bb4c9cd0040346e4241dd63ba44401eb02
Oct 20 11:05:53 host1 systemd[1]: Started Ceph nfs.bar.host1 for 9cfc6bda-89b5-413b-830f-01c8ddad63ec.
Oct 20 11:05:55 host1 conmon[17255]: 20/10/2020 17:05:55 : epoch 5f8f18f3 : host1 : ganesha.nfsd-1[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 3.3
Oct 20 11:06:05 host1 conmon[17255]: 20/10/2020 17:06:05 : epoch 5f8f18f3 : host1 : ganesha.nfsd-1[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
Oct 20 11:06:05 host1 conmon[17255]: 20/10/2020 17:06:05 : epoch 5f8f18f3 : host1 : ganesha.nfsd-1[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
Oct 20 11:06:06 host1 conmon[17255]: 20/10/2020 17:06:06 : epoch 5f8f18f3 : host1 : ganesha.nfsd-1[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
Oct 20 11:06:11 host1 conmon[17255]: 20/10/2020 17:06:11 : epoch 5f8f18f3 : host1 : ganesha.nfsd-1[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 
Oct 20 11:06:33 host1 conmon[17255]: 20/10/2020 17:06:33 : epoch 5f8f18f3 : host1 : ganesha.nfsd-1[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
Oct 20 11:06:36 host1 conmon[17255]: 2020-10-20T17:06:36.141+0000 7f878de25400 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-admin/keyring: (2) No such file or directory
Oct 20 11:06:36 host1 conmon[17255]: 2020-10-20T17:06:36.141+0000 7f878de25400 -1 AuthRegistry(0x55897c3c5290) no keyring found at /var/lib/ceph/radosgw/ceph-admin/keyring, disabling cephx
Oct 20 11:06:36 host1 conmon[17255]: 2020-10-20T17:06:36.141+0000 7f878de25400 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-admin/keyring: (2) No such file or directory
Oct 20 11:06:36 host1 conmon[17255]: 2020-10-20T17:06:36.141+0000 7f878de25400 -1 AuthRegistry(0x7fffe7f93160) no keyring found at /var/lib/ceph/radosgw/ceph-admin/keyring, disabling cephx
Oct 20 11:06:36 host1 conmon[17255]: 2020-10-20T17:06:36.145+0000 7f8723fff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
Oct 20 11:06:36 host1 conmon[17255]: 2020-10-20T17:06:36.145+0000 7f8722ffd700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
Oct 20 11:06:36 host1 conmon[17255]: 2020-10-20T17:06:36.145+0000 7f87237fe700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
Oct 20 11:06:36 host1 conmon[17255]: 2020-10-20T17:06:36.145+0000 7f878de25400 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication
Oct 20 11:06:36 host1 conmon[17255]: failed to fetch mon config (--no-mon-config to skip)
Oct 20 11:06:54 host1 systemd[1]: ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec@nfs.bar.host1.service: Main process exited, code=exited, status=1/FAILURE
Oct 20 11:06:54 host1 bash[17441]: Error: Failed to evict container: "": Failed to find container "ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec-nfs.bar.host1-grace-remove" in state: no container with name or ID ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec-nfs.bar.host1-grace-remove found: no such container
Oct 20 11:06:55 host1 bash[17487]: Error: no container with ID or name "ceph-9cfc6bda-89b5-413b-830f-01c8ddad63ec-nfs.bar.host1-grace-remove" found: no such container

how can we fix this for octopus?