Project

General

Profile

Bug #45726 » get_auth_error_in_mgr.log

Manager log with error - Juan Miguel Olmo Martínez, 05/27/2020 01:19 PM

 
-- Logs begin at Wed 2020-05-27 13:00:39 UTC, end at Wed 2020-05-27 13:17:27 UTC. --
May 27 13:09:37 ceph-node-00.cephlab.com systemd[1]: Starting Ceph mgr.ceph-node-00.cephlab.com.mfduko for 38ee6788-a01b-11ea-9d0e-52540095cc48...
May 27 13:09:37 ceph-node-00.cephlab.com podman[13059]: Error: no container with name or ID ceph-38ee6788-a01b-11ea-9d0e-52540095cc48-mgr.ceph-node-00.cephlab.com.mfduko found: no such container
May 27 13:09:37 ceph-node-00.cephlab.com systemd[1]: Started Ceph mgr.ceph-node-00.cephlab.com.mfduko for 38ee6788-a01b-11ea-9d0e-52540095cc48.
May 27 13:09:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:38.295+0000 7f2581b57040 0 set uid:gid to 167:167 (ceph:ceph)
May 27 13:09:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:38.295+0000 7f2581b57040 0 ceph version 16.0.0-1574-gacefe44a49 (acefe44a499324185088c767287ccfe628743877) pacific (dev), process ceph-mgr, pid 1
May 27 13:09:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:38.295+0000 7f2581b57040 0 pidfile_write: ignore empty --pid-file
May 27 13:09:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:38.380+0000 7f2581b57040 1 mgr[py] Loading python module 'alerts'
May 27 13:09:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:38.511+0000 7f2581b57040 1 mgr[py] Loading python module 'balancer'
May 27 13:09:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:38.619+0000 7f2581b57040 1 mgr[py] Loading python module 'cephadm'
May 27 13:09:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:38.880+0000 7f2581b57040 1 mgr[py] Loading python module 'crash'
May 27 13:09:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:38.972+0000 7f2581b57040 1 mgr[py] Loading python module 'dashboard'
May 27 13:09:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:39.668+0000 7f2581b57040 1 mgr[py] Loading python module 'devicehealth'
May 27 13:09:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:39.754+0000 7f2581b57040 1 mgr[py] Loading python module 'diskprediction_cloud'
May 27 13:09:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:39.944+0000 7f2581b57040 1 mgr[py] Loading python module 'diskprediction_local'
May 27 13:09:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:40.143+0000 7f2581b57040 1 mgr[py] Loading python module 'hello'
May 27 13:09:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:40.296+0000 7f2581b57040 1 mgr[py] Loading python module 'influx'
May 27 13:09:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:40.380+0000 7f2581b57040 1 mgr[py] Loading python module 'insights'
May 27 13:09:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:40.463+0000 7f2581b57040 1 mgr[py] Loading python module 'iostat'
May 27 13:09:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:40.528+0000 7f2581b57040 1 mgr[py] Loading python module 'k8sevents'
May 27 13:09:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:40.998+0000 7f2581b57040 1 mgr[py] Loading python module 'localpool'
May 27 13:09:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:41.064+0000 7f2581b57040 1 mgr[py] Loading python module 'orchestrator'
May 27 13:09:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:41.267+0000 7f2581b57040 1 mgr[py] Loading python module 'osd_perf_query'
May 27 13:09:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:41.342+0000 7f2581b57040 1 mgr[py] Loading python module 'osd_support'
May 27 13:09:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:41.421+0000 7f2581b57040 1 mgr[py] Loading python module 'pg_autoscaler'
May 27 13:09:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:41.529+0000 7f2581b57040 1 mgr[py] Loading python module 'progress'
May 27 13:09:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:41.631+0000 7f2581b57040 1 mgr[py] Loading python module 'prometheus'
May 27 13:09:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:42.121+0000 7f2581b57040 1 mgr[py] Loading python module 'rbd_support'
May 27 13:09:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:42.237+0000 7f2581b57040 1 mgr[py] Loading python module 'restful'
May 27 13:09:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:42.518+0000 7f2581b57040 1 mgr[py] Loading python module 'rook'
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:43.084+0000 7f2581b57040 -1 mgr[py] Module not found: 'rook'
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:43.084+0000 7f2581b57040 -1 mgr[py] Traceback (most recent call last):
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/__init__.py", line 2, in <module>
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: from .module import RookOrchestrator
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/module.py", line 36, in <module>
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: from .rook_cluster import RookCluster
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/rook_cluster.py", line 39, in <module>
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: from .rook_client.ceph import cephfilesystem as cfs
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: ModuleNotFoundError: No module named 'rook.rook_client'
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:43.093+0000 7f2581b57040 -1 mgr[py] Class not found in module 'rook'
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:43.093+0000 7f2581b57040 -1 mgr[py] Error loading module 'rook': (2) No such file or directory
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:43.093+0000 7f2581b57040 1 mgr[py] Loading python module 'selftest'
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:43.172+0000 7f2581b57040 1 mgr[py] Loading python module 'status'
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:43.282+0000 7f2581b57040 1 mgr[py] Loading python module 'telegraf'
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:43.375+0000 7f2581b57040 1 mgr[py] Loading python module 'telemetry'
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:43.656+0000 7f2581b57040 1 mgr[py] Loading python module 'test_orchestrator'
May 27 13:09:43 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:43.862+0000 7f2581b57040 1 mgr[py] Loading python module 'volumes'
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.215+0000 7f2581b57040 1 mgr[py] Loading python module 'zabbix'
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.290+0000 7f2581b57040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.290+0000 7f256ee50700 0 ms_deliver_dispatch: unhandled message 0x563bccb47a20 mon_map magic: 0 v1 from mon.0 v2:192.168.122.194:3300/0
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.309+0000 7f256ee50700 1 mgr handle_mgr_map Activating!
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.309+0000 7f256ee50700 1 mgr handle_mgr_map I am now activating
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.315+0000 7f255cd7b700 1 mgr[py] Upgrading module configuration for Mimic
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.318+0000 7f255cd7b700 0 [balancer DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.318+0000 7f255cd7b700 1 mgr load Constructed class from module: balancer
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.318+0000 7f255cd7b700 0 [crash DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.318+0000 7f255cd7b700 1 mgr load Constructed class from module: crash
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.320+0000 7f255cd7b700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.320+0000 7f255cd7b700 1 mgr load Constructed class from module: devicehealth
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.321+0000 7f255cd7b700 0 [iostat DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.321+0000 7f255cd7b700 1 mgr load Constructed class from module: iostat
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.321+0000 7f255cd7b700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.322+0000 7f255cd7b700 1 mgr load Constructed class from module: orchestrator
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.323+0000 7f255cd7b700 0 [osd_support DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.323+0000 7f255cd7b700 1 mgr load Constructed class from module: osd_support
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.323+0000 7f255cd7b700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.323+0000 7f255cd7b700 1 mgr load Constructed class from module: pg_autoscaler
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.327+0000 7f255cd7b700 0 [progress DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.327+0000 7f255cd7b700 1 mgr load Constructed class from module: progress
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.328+0000 7f255cd7b700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.339+0000 7f255cd7b700 1 mgr load Constructed class from module: rbd_support
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.339+0000 7f255cd7b700 0 [restful DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.339+0000 7f255cd7b700 1 mgr load Constructed class from module: restful
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.339+0000 7f255cd7b700 0 [status DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.339+0000 7f255cd7b700 1 mgr load Constructed class from module: status
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.340+0000 7f255cd7b700 0 [telemetry DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.340+0000 7f255cd7b700 1 mgr load Constructed class from module: telemetry
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.340+0000 7f255cd7b700 0 [volumes DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.342+0000 7f2541745700 0 [restful WARNING root] server not running: no certificate configured
May 27 13:09:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:44.616+0000 7f255cd7b700 1 mgr load Constructed class from module: volumes
May 27 13:09:46 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:46.313+0000 7f2550f64700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:09:48 ceph-node-00.cephlab.com bash[13093]: ignoring --setuser ceph since I am not root
May 27 13:09:48 ceph-node-00.cephlab.com bash[13093]: ignoring --setgroup ceph since I am not root
May 27 13:09:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:48.083+0000 7fb758734040 0 ceph version 16.0.0-1574-gacefe44a49 (acefe44a499324185088c767287ccfe628743877) pacific (dev), process ceph-mgr, pid 1
May 27 13:09:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:48.084+0000 7fb758734040 0 pidfile_write: ignore empty --pid-file
May 27 13:09:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:48.199+0000 7fb758734040 1 mgr[py] Loading python module 'alerts'
May 27 13:09:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:48.319+0000 7fb758734040 1 mgr[py] Loading python module 'balancer'
May 27 13:09:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:48.402+0000 7fb758734040 1 mgr[py] Loading python module 'cephadm'
May 27 13:09:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:48.670+0000 7fb758734040 1 mgr[py] Loading python module 'crash'
May 27 13:09:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:48.763+0000 7fb758734040 1 mgr[py] Loading python module 'dashboard'
May 27 13:09:49 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:49.560+0000 7fb758734040 1 mgr[py] Loading python module 'devicehealth'
May 27 13:09:49 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:49.644+0000 7fb758734040 1 mgr[py] Loading python module 'diskprediction_cloud'
May 27 13:09:49 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:49.829+0000 7fb758734040 1 mgr[py] Loading python module 'diskprediction_local'
May 27 13:09:49 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:49.976+0000 7fb758734040 1 mgr[py] Loading python module 'hello'
May 27 13:09:50 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:50.078+0000 7fb758734040 1 mgr[py] Loading python module 'influx'
May 27 13:09:50 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:50.147+0000 7fb758734040 1 mgr[py] Loading python module 'insights'
May 27 13:09:50 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:50.223+0000 7fb758734040 1 mgr[py] Loading python module 'iostat'
May 27 13:09:50 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:50.293+0000 7fb758734040 1 mgr[py] Loading python module 'k8sevents'
May 27 13:09:50 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:50.745+0000 7fb758734040 1 mgr[py] Loading python module 'localpool'
May 27 13:09:50 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:50.819+0000 7fb758734040 1 mgr[py] Loading python module 'orchestrator'
May 27 13:09:51 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:51.024+0000 7fb758734040 1 mgr[py] Loading python module 'osd_perf_query'
May 27 13:09:51 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:51.103+0000 7fb758734040 1 mgr[py] Loading python module 'osd_support'
May 27 13:09:51 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:51.173+0000 7fb758734040 1 mgr[py] Loading python module 'pg_autoscaler'
May 27 13:09:51 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:51.303+0000 7fb758734040 1 mgr[py] Loading python module 'progress'
May 27 13:09:51 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:51.430+0000 7fb758734040 1 mgr[py] Loading python module 'prometheus'
May 27 13:09:51 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:51.928+0000 7fb758734040 1 mgr[py] Loading python module 'rbd_support'
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:52.076+0000 7fb758734040 1 mgr[py] Loading python module 'restful'
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:52.395+0000 7fb758734040 1 mgr[py] Loading python module 'rook'
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:52.960+0000 7fb758734040 -1 mgr[py] Module not found: 'rook'
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:52.960+0000 7fb758734040 -1 mgr[py] Traceback (most recent call last):
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/__init__.py", line 2, in <module>
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: from .module import RookOrchestrator
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/module.py", line 36, in <module>
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: from .rook_cluster import RookCluster
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/rook_cluster.py", line 39, in <module>
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: from .rook_client.ceph import cephfilesystem as cfs
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: ModuleNotFoundError: No module named 'rook.rook_client'
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:52.967+0000 7fb758734040 -1 mgr[py] Class not found in module 'rook'
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:52.967+0000 7fb758734040 -1 mgr[py] Error loading module 'rook': (2) No such file or directory
May 27 13:09:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:52.967+0000 7fb758734040 1 mgr[py] Loading python module 'selftest'
May 27 13:09:53 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:53.038+0000 7fb758734040 1 mgr[py] Loading python module 'status'
May 27 13:09:53 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:53.130+0000 7fb758734040 1 mgr[py] Loading python module 'telegraf'
May 27 13:09:53 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:53.218+0000 7fb758734040 1 mgr[py] Loading python module 'telemetry'
May 27 13:09:53 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:53.490+0000 7fb758734040 1 mgr[py] Loading python module 'test_orchestrator'
May 27 13:09:53 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:53.656+0000 7fb758734040 1 mgr[py] Loading python module 'volumes'
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.008+0000 7fb758734040 1 mgr[py] Loading python module 'zabbix'
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.080+0000 7fb758734040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.082+0000 7fb745c3f700 0 ms_deliver_dispatch: unhandled message 0x55cc91fcba20 mon_map magic: 0 v1 from mon.0 v2:192.168.122.194:3300/0
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.162+0000 7fb745c3f700 1 mgr handle_mgr_map Activating!
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.163+0000 7fb745c3f700 1 mgr handle_mgr_map I am now activating
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.186+0000 7fb733b6a700 1 mgr[py] Upgrading module configuration for Mimic
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.189+0000 7fb733b6a700 0 [balancer DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.189+0000 7fb733b6a700 1 mgr load Constructed class from module: balancer
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.191+0000 7fb733b6a700 0 [cephadm DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.200+0000 7fb733b6a700 1 mgr load Constructed class from module: cephadm
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.201+0000 7fb733b6a700 0 [crash DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.201+0000 7fb733b6a700 1 mgr load Constructed class from module: crash
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.202+0000 7fb727d53700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.204+0000 7fb733b6a700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.204+0000 7fb733b6a700 1 mgr load Constructed class from module: devicehealth
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.204+0000 7fb733b6a700 0 [iostat DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.204+0000 7fb733b6a700 1 mgr load Constructed class from module: iostat
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.205+0000 7fb733b6a700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.205+0000 7fb733b6a700 1 mgr load Constructed class from module: orchestrator
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.206+0000 7fb733b6a700 0 [osd_support DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.206+0000 7fb733b6a700 1 mgr load Constructed class from module: osd_support
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.207+0000 7fb733b6a700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.207+0000 7fb733b6a700 1 mgr load Constructed class from module: pg_autoscaler
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.209+0000 7fb733b6a700 0 [progress DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.209+0000 7fb733b6a700 1 mgr load Constructed class from module: progress
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.209+0000 7fb733b6a700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.226+0000 7fb733b6a700 1 mgr load Constructed class from module: rbd_support
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.226+0000 7fb733b6a700 0 [restful DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.226+0000 7fb733b6a700 1 mgr load Constructed class from module: restful
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.227+0000 7fb733b6a700 0 [status DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.227+0000 7fb733b6a700 1 mgr load Constructed class from module: status
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.228+0000 7fb733b6a700 0 [telemetry DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.228+0000 7fb733b6a700 1 mgr load Constructed class from module: telemetry
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.229+0000 7fb7154ae700 0 [restful WARNING root] server not running: no certificate configured
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.229+0000 7fb733b6a700 0 [volumes DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:09:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:54.237+0000 7fb733b6a700 1 mgr load Constructed class from module: volumes
May 27 13:09:56 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:56.110+0000 7fb728d55700 0 log_channel(audit) log [DBG] : from='client.14133 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:09:56 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:56.202+0000 7fb727d53700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:09:56 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:56.858+0000 7fb728d55700 0 log_channel(audit) log [DBG] : from='client.14135 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:09:56 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:56.859+0000 7fb733b6a700 0 log_channel(cephadm) log [INF] : Generating ssh key...
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: Generating public/private rsa key pair.
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: Your identification has been saved in /tmp/tmp3l_4e3v5/key.
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: Your public key has been saved in /tmp/tmp3l_4e3v5/key.pub.
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: The key fingerprint is:
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: SHA256:zpmlf/6FBd7tnm2X1pp6pjTNeFNveUdJIk6BcelZ9ek ceph-38ee6788-a01b-11ea-9d0e-52540095cc48
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: The key's randomart image is:
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: +---[RSA 3072]----+
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: | .oo. .. |
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: | ..... o|
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: | .oo..o.|
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: | oo..+oo|
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: | S .. .E*|
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: | o = + *o|
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: | * + *.X|
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: | . ..ooOO|
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: | .o+*=++|
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: +----[SHA256]-----+
May 27 13:09:57 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:57.800+0000 7fb728d55700 0 log_channel(audit) log [DBG] : from='client.14137 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:09:58 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:58.202+0000 7fb727d53700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:09:58 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:58.520+0000 7fb728d55700 0 log_channel(audit) log [DBG] : from='client.14139 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "ceph-node-00.cephlab.com", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:09:58 ceph-node-00.cephlab.com bash[13093]: Warning: Permanently added 'ceph-node-00.cephlab.com,192.168.122.194' (ECDSA) to the list of known hosts.
May 27 13:09:59 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:09:59.196+0000 7fb733b6a700 0 log_channel(cephadm) log [INF] : Added host ceph-node-00.cephlab.com
May 27 13:10:00 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:00.185+0000 7fb728d55700 0 log_channel(audit) log [DBG] : from='client.14141 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:10:00 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:00.186+0000 7fb733b6a700 0 log_channel(cephadm) log [INF] : Saving service mon spec with placement count:5
May 27 13:10:00 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:00.202+0000 7fb727d53700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:10:01 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:01.234+0000 7fb728d55700 0 log_channel(audit) log [DBG] : from='client.14143 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:10:01 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:01.235+0000 7fb733b6a700 0 log_channel(cephadm) log [INF] : Saving service mgr spec with placement count:2
May 27 13:10:02 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:02.068+0000 7fb728d55700 0 log_channel(audit) log [DBG] : from='client.14145 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:10:02 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:02.068+0000 7fb733b6a700 0 log_channel(cephadm) log [INF] : Saving service crash spec with placement *
May 27 13:10:02 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:02.203+0000 7fb727d53700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:10:03 ceph-node-00.cephlab.com bash[13093]: debug ignoring --setuser ceph since I am not root
May 27 13:10:03 ceph-node-00.cephlab.com bash[13093]: ignoring --setgroup ceph since I am not root
May 27 13:10:03 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:03.163+0000 7f94a0bb1040 0 ceph version 16.0.0-1574-gacefe44a49 (acefe44a499324185088c767287ccfe628743877) pacific (dev), process ceph-mgr, pid 1
May 27 13:10:03 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:03.164+0000 7f94a0bb1040 0 pidfile_write: ignore empty --pid-file
May 27 13:10:03 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:03.301+0000 7f94a0bb1040 1 mgr[py] Loading python module 'alerts'
May 27 13:10:03 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:03.424+0000 7f94a0bb1040 1 mgr[py] Loading python module 'balancer'
May 27 13:10:03 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:03.509+0000 7f94a0bb1040 1 mgr[py] Loading python module 'cephadm'
May 27 13:10:03 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:03.763+0000 7f94a0bb1040 1 mgr[py] Loading python module 'crash'
May 27 13:10:03 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:03.850+0000 7f94a0bb1040 1 mgr[py] Loading python module 'dashboard'
May 27 13:10:04 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:04.573+0000 7f94a0bb1040 1 mgr[py] Loading python module 'devicehealth'
May 27 13:10:04 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:04.653+0000 7f94a0bb1040 1 mgr[py] Loading python module 'diskprediction_cloud'
May 27 13:10:04 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:04.838+0000 7f94a0bb1040 1 mgr[py] Loading python module 'diskprediction_local'
May 27 13:10:04 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:04.997+0000 7f94a0bb1040 1 mgr[py] Loading python module 'hello'
May 27 13:10:05 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:05.100+0000 7f94a0bb1040 1 mgr[py] Loading python module 'influx'
May 27 13:10:05 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:05.168+0000 7f94a0bb1040 1 mgr[py] Loading python module 'insights'
May 27 13:10:05 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:05.245+0000 7f94a0bb1040 1 mgr[py] Loading python module 'iostat'
May 27 13:10:05 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:05.313+0000 7f94a0bb1040 1 mgr[py] Loading python module 'k8sevents'
May 27 13:10:05 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:05.854+0000 7f94a0bb1040 1 mgr[py] Loading python module 'localpool'
May 27 13:10:05 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:05.942+0000 7f94a0bb1040 1 mgr[py] Loading python module 'orchestrator'
May 27 13:10:06 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:06.196+0000 7f94a0bb1040 1 mgr[py] Loading python module 'osd_perf_query'
May 27 13:10:06 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:06.296+0000 7f94a0bb1040 1 mgr[py] Loading python module 'osd_support'
May 27 13:10:06 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:06.379+0000 7f94a0bb1040 1 mgr[py] Loading python module 'pg_autoscaler'
May 27 13:10:06 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:06.487+0000 7f94a0bb1040 1 mgr[py] Loading python module 'progress'
May 27 13:10:06 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:06.583+0000 7f94a0bb1040 1 mgr[py] Loading python module 'prometheus'
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:07.002+0000 7f94a0bb1040 1 mgr[py] Loading python module 'rbd_support'
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:07.120+0000 7f94a0bb1040 1 mgr[py] Loading python module 'restful'
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:07.409+0000 7f94a0bb1040 1 mgr[py] Loading python module 'rook'
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:07.976+0000 7f94a0bb1040 -1 mgr[py] Module not found: 'rook'
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:07.976+0000 7f94a0bb1040 -1 mgr[py] Traceback (most recent call last):
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/__init__.py", line 2, in <module>
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: from .module import RookOrchestrator
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/module.py", line 36, in <module>
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: from .rook_cluster import RookCluster
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/rook_cluster.py", line 39, in <module>
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: from .rook_client.ceph import cephfilesystem as cfs
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: ModuleNotFoundError: No module named 'rook.rook_client'
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:07.984+0000 7f94a0bb1040 -1 mgr[py] Class not found in module 'rook'
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:07.984+0000 7f94a0bb1040 -1 mgr[py] Error loading module 'rook': (2) No such file or directory
May 27 13:10:07 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:07.984+0000 7f94a0bb1040 1 mgr[py] Loading python module 'selftest'
May 27 13:10:08 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:08.056+0000 7f94a0bb1040 1 mgr[py] Loading python module 'status'
May 27 13:10:08 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:08.156+0000 7f94a0bb1040 1 mgr[py] Loading python module 'telegraf'
May 27 13:10:08 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:08.250+0000 7f94a0bb1040 1 mgr[py] Loading python module 'telemetry'
May 27 13:10:08 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:08.522+0000 7f94a0bb1040 1 mgr[py] Loading python module 'test_orchestrator'
May 27 13:10:08 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:08.677+0000 7f94a0bb1040 1 mgr[py] Loading python module 'volumes'
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.050+0000 7f94a0bb1040 1 mgr[py] Loading python module 'zabbix'
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.123+0000 7f94a0bb1040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.126+0000 7f948e0bc700 0 ms_deliver_dispatch: unhandled message 0x56024c10b760 mon_map magic: 0 v1 from mon.0 v2:192.168.122.194:3300/0
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.205+0000 7f948e0bc700 1 mgr handle_mgr_map Activating!
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.205+0000 7f948e0bc700 1 mgr handle_mgr_map I am now activating
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.247+0000 7f947bfe7700 0 [balancer DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.248+0000 7f947bfe7700 1 mgr load Constructed class from module: balancer
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.249+0000 7f947bfe7700 0 [cephadm DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.261+0000 7f947bfe7700 1 mgr load Constructed class from module: cephadm
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.262+0000 7f947bfe7700 0 [crash DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.262+0000 7f947bfe7700 1 mgr load Constructed class from module: crash
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.264+0000 7f947bfe7700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.264+0000 7f947bfe7700 1 mgr load Constructed class from module: devicehealth
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.265+0000 7f947bfe7700 0 [iostat DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.265+0000 7f947bfe7700 1 mgr load Constructed class from module: iostat
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.266+0000 7f947bfe7700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.266+0000 7f947bfe7700 1 mgr load Constructed class from module: orchestrator
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.268+0000 7f947bfe7700 0 [osd_support DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.268+0000 7f947bfe7700 1 mgr load Constructed class from module: osd_support
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.273+0000 7f947bfe7700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.273+0000 7f947bfe7700 1 mgr load Constructed class from module: pg_autoscaler
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.274+0000 7f947bfe7700 0 [progress DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.274+0000 7f947bfe7700 1 mgr load Constructed class from module: progress
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.275+0000 7f947bfe7700 0 [prometheus DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.275+0000 7f947bfe7700 1 mgr load Constructed class from module: prometheus
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.276+0000 7f947bfe7700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: [27/May/2020:13:10:09] ENGINE Bus STARTING
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.304+0000 7f947bfe7700 1 mgr load Constructed class from module: rbd_support
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.304+0000 7f947bfe7700 0 [restful DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.304+0000 7f947bfe7700 1 mgr load Constructed class from module: restful
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.305+0000 7f947bfe7700 0 [status DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.305+0000 7f947bfe7700 1 mgr load Constructed class from module: status
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.305+0000 7f947bfe7700 0 [telemetry DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.305+0000 7f947bfe7700 1 mgr load Constructed class from module: telemetry
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.306+0000 7f947bfe7700 0 [volumes DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.308+0000 7f945a48c700 0 [restful WARNING root] server not running: no certificate configured
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:09.322+0000 7f947bfe7700 1 mgr load Constructed class from module: volumes
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: Warning: Permanently added 'ceph-node-00.cephlab.com,192.168.122.194' (ECDSA) to the list of known hosts.
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: CherryPy Checker:
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: The Application mounted at '' has an empty config.
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: [27/May/2020:13:10:09] ENGINE Serving on http://:::9283
May 27 13:10:09 ceph-node-00.cephlab.com bash[13093]: [27/May/2020:13:10:09] ENGINE Bus STARTED
May 27 13:10:10 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:10.285+0000 7f94711d2700 0 log_channel(audit) log [DBG] : from='client.14151 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:10:10 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:10.285+0000 7f947bfe7700 0 log_channel(cephadm) log [INF] : Saving service prometheus spec with placement count:1
May 27 13:10:11 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:11.223+0000 7f94701d0700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:10:11 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:11.358+0000 7f94711d2700 0 log_channel(audit) log [DBG] : from='client.14157 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:10:11 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:11.359+0000 7f947bfe7700 0 log_channel(cephadm) log [INF] : Saving service grafana spec with placement count:1
May 27 13:10:12 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:12.162+0000 7f94711d2700 0 log_channel(audit) log [DBG] : from='client.14159 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:10:12 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:12.163+0000 7f947bfe7700 0 log_channel(cephadm) log [INF] : Saving service node-exporter spec with placement *
May 27 13:10:12 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:12.922+0000 7f94711d2700 0 log_channel(audit) log [DBG] : from='client.14161 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:10:12 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:12.923+0000 7f947bfe7700 0 log_channel(cephadm) log [INF] : Saving service alertmanager spec with placement count:1
May 27 13:10:13 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:13.223+0000 7f94701d0700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:10:15 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:15.223+0000 7f94701d0700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:10:15 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:15.389+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Deploying daemon crash.ceph-node-00 on ceph-node-00.cephlab.com
May 27 13:10:16 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:16.825+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Deploying daemon prometheus.ceph-node-00 on ceph-node-00.cephlab.com
May 27 13:10:17 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:17.223+0000 7f94701d0700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:10:19 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:19.223+0000 7f94701d0700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:10:21 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:21.223+0000 7f94701d0700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:10:23 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:23.223+0000 7f94701d0700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:10:25 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:25.223+0000 7f94701d0700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:10:27 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:27.224+0000 7f94701d0700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:10:29 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:29.234+0000 7f94701d0700 1 mgr.server send_report Giving up on OSDs that haven't reported yet, sending potentially incomplete PG state to mon
May 27 13:10:29 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:29.234+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v3: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:31.235+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v4: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:31.399+0000 7f94711d2700 0 log_channel(audit) log [DBG] : from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:31.399+0000 7f947bfe7700 -1 mgr.server reply reply (95) Operation not supported Module 'dashboard' is not enabled (required by command 'dashboard set-grafana-api-ssl-verify'): use `ceph mgr module enable dashboard` to enable it
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:31.410+0000 7f94681c0700 0 [cephadm ERROR root] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False}): dashboard set-grafana-api-ssl-verify failed: Module 'dashboard' is not enabled (required by command 'dashboard set-grafana-api-ssl-verify'): use `ceph mgr module enable dashboard` to enable it
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: Traceback (most recent call last):
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1775, in _apply_all_services
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: if self._apply_service(spec):
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1746, in _apply_service
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: create_func(daemon_id, host) # type: ignore
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/services/monitoring.py", line 14, in create
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: return self.mgr._create_daemon('grafana', daemon_id, host)
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1551, in _create_daemon
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: cephadm_config, deps = self.grafana_service.generate_config()
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/services/monitoring.py", line 76, in generate_config
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: 'value': 'false',
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/mgr_module.py", line 1096, in check_mon_command
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: raise MonCommandFailed(f'{cmd_dict["prefix"]} failed: {r.stderr}')
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: mgr_module.MonCommandFailed: dashboard set-grafana-api-ssl-verify failed: Module 'dashboard' is not enabled (required by command 'dashboard set-grafana-api-ssl-verify'): use `ceph mgr module enable dashboard` to enable it
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:31.411+0000 7f94681c0700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False}): dashboard set-grafana-api-ssl-verify failed: Module 'dashboard' is not enabled (required by command 'dashboard set-grafana-api-ssl-verify'): use `ceph mgr module enable dashboard` to enable it
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: Traceback (most recent call last):
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1775, in _apply_all_services
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: if self._apply_service(spec):
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1746, in _apply_service
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: create_func(daemon_id, host) # type: ignore
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/services/monitoring.py", line 14, in create
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: return self.mgr._create_daemon('grafana', daemon_id, host)
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1551, in _create_daemon
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: cephadm_config, deps = self.grafana_service.generate_config()
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/services/monitoring.py", line 76, in generate_config
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: 'value': 'false',
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/mgr_module.py", line 1096, in check_mon_command
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: raise MonCommandFailed(f'{cmd_dict["prefix"]} failed: {r.stderr}')
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: mgr_module.MonCommandFailed: dashboard set-grafana-api-ssl-verify failed: Module 'dashboard' is not enabled (required by command 'dashboard set-grafana-api-ssl-verify'): use `ceph mgr module enable dashboard` to enable it
May 27 13:10:31 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:31.412+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Deploying daemon node-exporter.ceph-node-00 on ceph-node-00.cephlab.com
May 27 13:10:32 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:32.328+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Deploying daemon alertmanager.ceph-node-00 on ceph-node-00.cephlab.com
May 27 13:10:33 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:33.235+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v5: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:35 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:35.236+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v6: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:37 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:37.237+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v7: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:37 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:10:37] "GET /metrics HTTP/1.1" 200 36256 "" "Prometheus/2.18.1"
May 27 13:10:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:39.237+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v8: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:41.237+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v9: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:43 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:43.238+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v10: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:44.255+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Deploying daemon grafana.ceph-node-00 on ceph-node-00.cephlab.com
May 27 13:10:45 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:45.238+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v11: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:47 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:47.238+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v12: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:47 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:10:47] "GET /metrics HTTP/1.1" 200 36268 "" "Prometheus/2.18.1"
May 27 13:10:49 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:49.239+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v13: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:51 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:51.239+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v14: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:53 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:53.239+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v15: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:55 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:55.239+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v16: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:57 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:57.240+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v17: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:10:57 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:10:57] "GET /metrics HTTP/1.1" 200 36268 "" "Prometheus/2.18.1"
May 27 13:10:59 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:10:59.240+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v18: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:01 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:01.240+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v19: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:03 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:03.241+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v20: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:05 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:05.241+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v21: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:07 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:07.241+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v22: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:07 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:11:07] "GET /metrics HTTP/1.1" 200 36266 "" "Prometheus/2.18.1"
May 27 13:11:09 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:09.241+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v23: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:10 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:10.851+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.ceph-node-00.cephlab.com (unknown last config time)...
May 27 13:11:10 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:10.854+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.ceph-node-00.cephlab.com on ceph-node-00.cephlab.com
May 27 13:11:11 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:11.242+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v24: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:11 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:11.817+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Reconfiguring prometheus.ceph-node-00 (dependencies changed)...
May 27 13:11:11 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:11.819+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon prometheus.ceph-node-00 on ceph-node-00.cephlab.com
May 27 13:11:13 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:13.242+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v25: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:13 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:13.994+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Reconfiguring alertmanager.ceph-node-00 (dependencies changed)...
May 27 13:11:13 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:13.995+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon alertmanager.ceph-node-00 on ceph-node-00.cephlab.com
May 27 13:11:15 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:15.243+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v26: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:15 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:15.761+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Reconfiguring mgr.ceph-node-00.cephlab.com.mfduko (unknown last config time)...
May 27 13:11:15 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:15.762+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mgr.ceph-node-00.cephlab.com.mfduko on ceph-node-00.cephlab.com
May 27 13:11:16 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:16.778+0000 7f94681c0700 0 log_channel(cephadm) log [INF] : Setting dashboard grafana config to https://ceph-node-00.cephlab.com:3000
May 27 13:11:17 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:17.242+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v27: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:19 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:19.242+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v28: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:21 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:21.243+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v29: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:23 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:23.244+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v30: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:25 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:25.244+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v31: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:27 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:27.245+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v32: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:27 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:11:27] "GET /metrics HTTP/1.1" 200 36267 "" "Prometheus/2.18.1"
May 27 13:11:29 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:29.245+0000 7f94701d0700 0 log_channel(cluster) log [DBG] : pgmap v33: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:11:30 ceph-node-00.cephlab.com bash[13093]: ignoring --setuser ceph since I am not root
May 27 13:11:30 ceph-node-00.cephlab.com bash[13093]: ignoring --setgroup ceph since I am not root
May 27 13:11:30 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:30.466+0000 7fdedb7e0040 0 ceph version 16.0.0-1574-gacefe44a49 (acefe44a499324185088c767287ccfe628743877) pacific (dev), process ceph-mgr, pid 1
May 27 13:11:30 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:30.467+0000 7fdedb7e0040 0 pidfile_write: ignore empty --pid-file
May 27 13:11:30 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:30.542+0000 7fdedb7e0040 1 mgr[py] Loading python module 'alerts'
May 27 13:11:30 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:30.674+0000 7fdedb7e0040 1 mgr[py] Loading python module 'balancer'
May 27 13:11:30 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:30.773+0000 7fdedb7e0040 1 mgr[py] Loading python module 'cephadm'
May 27 13:11:31 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:31.005+0000 7fdedb7e0040 1 mgr[py] Loading python module 'crash'
May 27 13:11:31 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:31.099+0000 7fdedb7e0040 1 mgr[py] Loading python module 'dashboard'
May 27 13:11:31 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:31.816+0000 7fdedb7e0040 1 mgr[py] Loading python module 'devicehealth'
May 27 13:11:31 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:31.911+0000 7fdedb7e0040 1 mgr[py] Loading python module 'diskprediction_cloud'
May 27 13:11:32 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:32.146+0000 7fdedb7e0040 1 mgr[py] Loading python module 'diskprediction_local'
May 27 13:11:32 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:32.353+0000 7fdedb7e0040 1 mgr[py] Loading python module 'hello'
May 27 13:11:32 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:32.490+0000 7fdedb7e0040 1 mgr[py] Loading python module 'influx'
May 27 13:11:32 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:32.582+0000 7fdedb7e0040 1 mgr[py] Loading python module 'insights'
May 27 13:11:32 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:32.676+0000 7fdedb7e0040 1 mgr[py] Loading python module 'iostat'
May 27 13:11:32 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:32.755+0000 7fdedb7e0040 1 mgr[py] Loading python module 'k8sevents'
May 27 13:11:33 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:33.224+0000 7fdedb7e0040 1 mgr[py] Loading python module 'localpool'
May 27 13:11:33 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:33.299+0000 7fdedb7e0040 1 mgr[py] Loading python module 'orchestrator'
May 27 13:11:33 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:33.514+0000 7fdedb7e0040 1 mgr[py] Loading python module 'osd_perf_query'
May 27 13:11:33 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:33.591+0000 7fdedb7e0040 1 mgr[py] Loading python module 'osd_support'
May 27 13:11:33 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:33.658+0000 7fdedb7e0040 1 mgr[py] Loading python module 'pg_autoscaler'
May 27 13:11:33 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:33.766+0000 7fdedb7e0040 1 mgr[py] Loading python module 'progress'
May 27 13:11:33 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:33.862+0000 7fdedb7e0040 1 mgr[py] Loading python module 'prometheus'
May 27 13:11:34 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:34.291+0000 7fdedb7e0040 1 mgr[py] Loading python module 'rbd_support'
May 27 13:11:34 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:34.403+0000 7fdedb7e0040 1 mgr[py] Loading python module 'restful'
May 27 13:11:34 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:34.692+0000 7fdedb7e0040 1 mgr[py] Loading python module 'rook'
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:35.374+0000 7fdedb7e0040 -1 mgr[py] Module not found: 'rook'
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:35.375+0000 7fdedb7e0040 -1 mgr[py] Traceback (most recent call last):
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/__init__.py", line 2, in <module>
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: from .module import RookOrchestrator
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/module.py", line 36, in <module>
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: from .rook_cluster import RookCluster
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/rook_cluster.py", line 39, in <module>
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: from .rook_client.ceph import cephfilesystem as cfs
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: ModuleNotFoundError: No module named 'rook.rook_client'
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:35.383+0000 7fdedb7e0040 -1 mgr[py] Class not found in module 'rook'
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:35.383+0000 7fdedb7e0040 -1 mgr[py] Error loading module 'rook': (2) No such file or directory
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:35.383+0000 7fdedb7e0040 1 mgr[py] Loading python module 'selftest'
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:35.479+0000 7fdedb7e0040 1 mgr[py] Loading python module 'status'
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:35.596+0000 7fdedb7e0040 1 mgr[py] Loading python module 'telegraf'
May 27 13:11:35 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:35.698+0000 7fdedb7e0040 1 mgr[py] Loading python module 'telemetry'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.003+0000 7fdedb7e0040 1 mgr[py] Loading python module 'test_orchestrator'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.184+0000 7fdedb7e0040 1 mgr[py] Loading python module 'volumes'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.551+0000 7fdedb7e0040 1 mgr[py] Loading python module 'zabbix'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.621+0000 7fdedb7e0040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.624+0000 7fdec8ceb700 0 ms_deliver_dispatch: unhandled message 0x55a98d6c1760 mon_map magic: 0 v1 from mon.0 v2:192.168.122.194:3300/0
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn e: '/usr/bin/ceph-mgr'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn 0: '/usr/bin/ceph-mgr'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn 1: '-n'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn 2: 'mgr.ceph-node-00.cephlab.com.mfduko'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn 3: '-f'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn 4: '--setuser'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn 5: 'ceph'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn 6: '--setgroup'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn 7: 'ceph'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn 8: '--default-log-to-file=false'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn 9: '--default-log-to-stderr=true'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn 10: '--default-log-stderr-prefix=debug '
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.625+0000 7fdec8ceb700 1 mgr respawn respawning with exe /usr/bin/ceph-mgr
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug ignoring --setuser ceph since I am not root
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: ignoring --setgroup ceph since I am not root
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.669+0000 7f6bd16a9040 0 ceph version 16.0.0-1574-gacefe44a49 (acefe44a499324185088c767287ccfe628743877) pacific (dev), process ceph-mgr, pid 1
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.669+0000 7f6bd16a9040 0 pidfile_write: ignore empty --pid-file
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.731+0000 7f6bd16a9040 1 mgr[py] Loading python module 'alerts'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.838+0000 7f6bd16a9040 1 mgr[py] Loading python module 'balancer'
May 27 13:11:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:36.925+0000 7f6bd16a9040 1 mgr[py] Loading python module 'cephadm'
May 27 13:11:37 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:37.164+0000 7f6bd16a9040 1 mgr[py] Loading python module 'crash'
May 27 13:11:37 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:37.256+0000 7f6bd16a9040 1 mgr[py] Loading python module 'dashboard'
May 27 13:11:37 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:37.960+0000 7f6bd16a9040 1 mgr[py] Loading python module 'devicehealth'
May 27 13:11:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:38.035+0000 7f6bd16a9040 1 mgr[py] Loading python module 'diskprediction_cloud'
May 27 13:11:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:38.216+0000 7f6bd16a9040 1 mgr[py] Loading python module 'diskprediction_local'
May 27 13:11:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:38.416+0000 7f6bd16a9040 1 mgr[py] Loading python module 'hello'
May 27 13:11:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:38.515+0000 7f6bd16a9040 1 mgr[py] Loading python module 'influx'
May 27 13:11:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:38.588+0000 7f6bd16a9040 1 mgr[py] Loading python module 'insights'
May 27 13:11:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:38.663+0000 7f6bd16a9040 1 mgr[py] Loading python module 'iostat'
May 27 13:11:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:38.739+0000 7f6bd16a9040 1 mgr[py] Loading python module 'k8sevents'
May 27 13:11:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:39.195+0000 7f6bd16a9040 1 mgr[py] Loading python module 'localpool'
May 27 13:11:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:39.262+0000 7f6bd16a9040 1 mgr[py] Loading python module 'orchestrator'
May 27 13:11:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:39.475+0000 7f6bd16a9040 1 mgr[py] Loading python module 'osd_perf_query'
May 27 13:11:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:39.553+0000 7f6bd16a9040 1 mgr[py] Loading python module 'osd_support'
May 27 13:11:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:39.618+0000 7f6bd16a9040 1 mgr[py] Loading python module 'pg_autoscaler'
May 27 13:11:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:39.714+0000 7f6bd16a9040 1 mgr[py] Loading python module 'progress'
May 27 13:11:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:39.810+0000 7f6bd16a9040 1 mgr[py] Loading python module 'prometheus'
May 27 13:11:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:40.248+0000 7f6bd16a9040 1 mgr[py] Loading python module 'rbd_support'
May 27 13:11:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:40.372+0000 7f6bd16a9040 1 mgr[py] Loading python module 'restful'
May 27 13:11:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:40.673+0000 7f6bd16a9040 1 mgr[py] Loading python module 'rook'
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:41.282+0000 7f6bd16a9040 -1 mgr[py] Module not found: 'rook'
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:41.282+0000 7f6bd16a9040 -1 mgr[py] Traceback (most recent call last):
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/__init__.py", line 2, in <module>
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: from .module import RookOrchestrator
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/module.py", line 36, in <module>
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: from .rook_cluster import RookCluster
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/rook/rook_cluster.py", line 39, in <module>
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: from .rook_client.ceph import cephfilesystem as cfs
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: ModuleNotFoundError: No module named 'rook.rook_client'
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:41.288+0000 7f6bd16a9040 -1 mgr[py] Class not found in module 'rook'
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:41.288+0000 7f6bd16a9040 -1 mgr[py] Error loading module 'rook': (2) No such file or directory
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:41.288+0000 7f6bd16a9040 1 mgr[py] Loading python module 'selftest'
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:41.363+0000 7f6bd16a9040 1 mgr[py] Loading python module 'status'
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:41.471+0000 7f6bd16a9040 1 mgr[py] Loading python module 'telegraf'
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:41.560+0000 7f6bd16a9040 1 mgr[py] Loading python module 'telemetry'
May 27 13:11:41 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:41.843+0000 7f6bd16a9040 1 mgr[py] Loading python module 'test_orchestrator'
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.006+0000 7f6bd16a9040 1 mgr[py] Loading python module 'volumes'
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.356+0000 7f6bd16a9040 1 mgr[py] Loading python module 'zabbix'
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.424+0000 7f6bd16a9040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.427+0000 7f6bbebb4700 0 ms_deliver_dispatch: unhandled message 0x55f539b17760 mon_map magic: 0 v1 from mon.0 v2:192.168.122.194:3300/0
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.503+0000 7f6bbebb4700 1 mgr handle_mgr_map Activating!
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.503+0000 7f6bbebb4700 1 mgr handle_mgr_map I am now activating
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.528+0000 7f6bacadf700 0 [balancer DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.528+0000 7f6bacadf700 1 mgr load Constructed class from module: balancer
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.531+0000 7f6bacadf700 0 [cephadm DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.550+0000 7f6bacadf700 1 mgr load Constructed class from module: cephadm
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.551+0000 7f6bacadf700 0 [crash DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.552+0000 7f6bacadf700 1 mgr load Constructed class from module: crash
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.553+0000 7f6bacadf700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.553+0000 7f6bacadf700 1 mgr load Constructed class from module: devicehealth
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.553+0000 7f6bacadf700 0 [iostat DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.553+0000 7f6bacadf700 1 mgr load Constructed class from module: iostat
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.556+0000 7f6bacadf700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.556+0000 7f6bacadf700 1 mgr load Constructed class from module: orchestrator
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.557+0000 7f6bacadf700 0 [osd_support DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.558+0000 7f6bacadf700 1 mgr load Constructed class from module: osd_support
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.558+0000 7f6bacadf700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.559+0000 7f6bacadf700 1 mgr load Constructed class from module: pg_autoscaler
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.559+0000 7f6bacadf700 0 [progress DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.559+0000 7f6bacadf700 1 mgr load Constructed class from module: progress
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.561+0000 7f6bacadf700 0 [prometheus DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.561+0000 7f6bacadf700 1 mgr load Constructed class from module: prometheus
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.569+0000 7f6bacadf700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: [27/May/2020:13:11:42] ENGINE Bus STARTING
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.589+0000 7f6bacadf700 1 mgr load Constructed class from module: rbd_support
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.590+0000 7f6bacadf700 0 [restful DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.590+0000 7f6bacadf700 1 mgr load Constructed class from module: restful
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.591+0000 7f6bacadf700 0 [status DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.591+0000 7f6bacadf700 1 mgr load Constructed class from module: status
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.591+0000 7f6bacadf700 0 [telemetry DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.591+0000 7f6bacadf700 1 mgr load Constructed class from module: telemetry
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.591+0000 7f6bacadf700 0 [volumes DEBUG root] setting log level based on debug_mgr: WARNING (1/5)
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.595+0000 7f6b8b785700 0 [restful WARNING root] server not running: no certificate configured
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: Warning: Permanently added 'ceph-node-00.cephlab.com,192.168.122.194' (ECDSA) to the list of known hosts.
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:42.614+0000 7f6bacadf700 1 mgr load Constructed class from module: volumes
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: CherryPy Checker:
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: The Application mounted at '' has an empty config.
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: [27/May/2020:13:11:42] ENGINE Serving on http://:::9283
May 27 13:11:42 ceph-node-00.cephlab.com bash[13093]: [27/May/2020:13:11:42] ENGINE Bus STARTED
May 27 13:11:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:44.509+0000 7f6ba0cc8700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:11:46 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:46.510+0000 7f6ba0cc8700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:11:47 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:11:47] "GET /metrics HTTP/1.1" 200 36270 "" "Prometheus/2.18.1"
May 27 13:11:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:48.511+0000 7f6ba0cc8700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:11:50 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:50.511+0000 7f6ba0cc8700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:11:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:52.511+0000 7f6ba0cc8700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:11:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:54.511+0000 7f6ba0cc8700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:11:56 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:56.511+0000 7f6ba0cc8700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:11:57 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:11:57] "GET /metrics HTTP/1.1" 200 36269 "" "Prometheus/2.18.1"
May 27 13:11:58 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:11:58.512+0000 7f6ba0cc8700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:12:00 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:00.512+0000 7f6ba0cc8700 1 mgr.server send_report Not sending PG status to monitor yet, waiting for OSDs
May 27 13:12:02 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:02.512+0000 7f6ba0cc8700 1 mgr.server send_report Giving up on OSDs that haven't reported yet, sending potentially incomplete PG state to mon
May 27 13:12:02 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:02.512+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v3: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:04 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:04.514+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v4: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:06 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:06.514+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v5: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:07 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:12:07] "GET /metrics HTTP/1.1" 200 36267 "" "Prometheus/2.18.1"
May 27 13:12:08 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:08.514+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v6: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:10 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:10.515+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v7: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:12 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:12.515+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v8: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:14 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:14.516+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v9: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:16 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:16.516+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v10: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:17 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:12:17] "GET /metrics HTTP/1.1" 200 36271 "" "Prometheus/2.18.1"
May 27 13:12:18 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:18.517+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v11: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:20 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:20.517+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v12: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:22 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:22.518+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v13: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:24 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:24.518+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v14: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:26 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:26.519+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v15: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:27 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:12:27] "GET /metrics HTTP/1.1" 200 36271 "" "Prometheus/2.18.1"
May 27 13:12:28 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:28.519+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v16: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:30 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:30.520+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v17: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:32 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:32.521+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v18: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:34 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:34.521+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v19: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:36.522+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v20: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:37 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:12:37] "GET /metrics HTTP/1.1" 200 36271 "" "Prometheus/2.18.1"
May 27 13:12:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:38.522+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v21: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:40.523+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v22: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:42.523+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v23: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:44.523+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v24: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:46 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:46.524+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v25: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:47 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:12:47] "GET /metrics HTTP/1.1" 200 36273 "" "Prometheus/2.18.1"
May 27 13:12:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:48.524+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v26: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:50 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:50.525+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v27: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:52.525+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v28: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:54.525+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v29: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:56 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:56.526+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v30: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:12:57 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:12:57] "GET /metrics HTTP/1.1" 200 36272 "" "Prometheus/2.18.1"
May 27 13:12:58 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:12:58.526+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v31: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:00 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:00.526+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v32: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:02 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:02.527+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v33: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:04 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:04.527+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v34: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:06 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:06.528+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v35: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:07 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:13:07] "GET /metrics HTTP/1.1" 200 36275 "" "Prometheus/2.18.1"
May 27 13:13:08 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:08.528+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v36: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:10 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:10.528+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v37: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:12 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:12.529+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v38: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:14 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:14.530+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v39: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:16 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:16.530+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v40: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:17 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:13:17] "GET /metrics HTTP/1.1" 200 36273 "" "Prometheus/2.18.1"
May 27 13:13:18 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:18.531+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v41: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:20 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:20.531+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v42: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:22 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:22.531+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v43: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:24 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:24.532+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v44: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:26 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:26.532+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v45: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:27 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:13:27] "GET /metrics HTTP/1.1" 200 36274 "" "Prometheus/2.18.1"
May 27 13:13:28 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:28.532+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v46: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:30 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:30.533+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v47: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:33 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:32.469+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v48: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:34 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:34.534+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v49: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:36.265+0000 7f6ba1cca700 0 log_channel(audit) log [DBG] : from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:13:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:36.534+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v50: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:37 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:13:37] "GET /metrics HTTP/1.1" 200 36275 "" "Prometheus/2.18.1"
May 27 13:13:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:38.535+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v51: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:40.535+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v52: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:42.536+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v53: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:44.536+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v54: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:44.638+0000 7f6ba1cca700 0 log_channel(audit) log [DBG] : from='client.14207 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "ceph-node-01.cephlab.com", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:13:44 ceph-node-00.cephlab.com bash[13093]: Warning: Permanently added 'ceph-node-01.cephlab.com,192.168.122.241' (ECDSA) to the list of known hosts.
May 27 13:13:45 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:45.318+0000 7f6bacadf700 0 log_channel(cephadm) log [INF] : Added host ceph-node-01.cephlab.com
May 27 13:13:46 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:46.537+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v55: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:47 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:13:47] "GET /metrics HTTP/1.1" 200 36276 "" "Prometheus/2.18.1"
May 27 13:13:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:48.536+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v56: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:50 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:50.537+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v57: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:51 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:51.388+0000 7f6ba1cca700 0 log_channel(audit) log [DBG] : from='client.14209 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:13:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:52.537+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v58: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:54.538+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v59: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:56 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:56.538+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v60: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:13:57 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:13:57] "GET /metrics HTTP/1.1" 200 36277 "" "Prometheus/2.18.1"
May 27 13:13:58 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:13:58.538+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v61: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:00 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:00.538+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v62: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:02 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:02.539+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v63: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:04 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:04.539+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v64: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:06 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:06.539+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v65: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:07 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:14:07] "GET /metrics HTTP/1.1" 200 36277 "" "Prometheus/2.18.1"
May 27 13:14:08 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:08.539+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v66: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:10 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:10.540+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v67: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:12 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:12.540+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v68: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:14 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:14.540+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v69: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:15 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:15.030+0000 7f6ba1cca700 0 log_channel(audit) log [DBG] : from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:14:16 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:16.541+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v70: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:17 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:14:17] "GET /metrics HTTP/1.1" 200 36276 "" "Prometheus/2.18.1"
May 27 13:14:18 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:18.541+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v71: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:20 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:20.542+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v72: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:22 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:22.542+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v73: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:23 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:23.389+0000 7f6b98cb8700 -1 ceph_set_health_checks check CEPHADM_REFRESH_FAILED detail item 0 not a [unicode] string
May 27 13:14:23 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:23.389+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v74: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:23 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:23.390+0000 7f6b98cb8700 -1 ceph_set_health_checks check CEPHADM_REFRESH_FAILED detail item 0 not a [unicode] string
May 27 13:14:23 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:23.405+0000 7f6b98cb8700 0 log_channel(cephadm) log [INF] : Deploying daemon crash.ceph-node-01 on ceph-node-01.cephlab.com
May 27 13:14:25 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:25.111+0000 7f6b98cb8700 0 log_channel(cephadm) log [INF] : Deploying daemon mgr.ceph-node-01.bmmxck on ceph-node-01.cephlab.com
May 27 13:14:25 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:25.390+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v75: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:26 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:26.400+0000 7f6b98cb8700 0 log_channel(cephadm) log [INF] : Deploying daemon mon.ceph-node-01 on ceph-node-01.cephlab.com
May 27 13:14:27 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:27.390+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v76: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:27 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:14:27] "GET /metrics HTTP/1.1" 200 36277 "" "Prometheus/2.18.1"
May 27 13:14:28 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:28.408+0000 7f6b98cb8700 0 log_channel(cephadm) log [INF] : Deploying daemon node-exporter.ceph-node-01 on ceph-node-01.cephlab.com
May 27 13:14:29 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:29.390+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v77: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:29 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:29.423+0000 7f6bbd3b1700 1 mgr finish mon failed to return metadata for mon.ceph-node-01: (2) No such file or directory
May 27 13:14:30 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:30.419+0000 7f6bbd3b1700 1 mgr finish mon failed to return metadata for mon.ceph-node-01: (22) Invalid argument
May 27 13:14:31 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:31.391+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v78: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:31 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:31.419+0000 7f6bbd3b1700 1 mgr finish mon failed to return metadata for mon.ceph-node-01: (22) Invalid argument
May 27 13:14:32 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:32.421+0000 7f6bbd3b1700 1 mgr finish mon failed to return metadata for mon.ceph-node-01: (22) Invalid argument
May 27 13:14:33 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:33.391+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v79: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:33 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:33.420+0000 7f6bbd3b1700 1 mgr finish mon failed to return metadata for mon.ceph-node-01: (22) Invalid argument
May 27 13:14:34 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:34.421+0000 7f6bbd3b1700 1 mgr finish mon failed to return metadata for mon.ceph-node-01: (22) Invalid argument
May 27 13:14:34 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:34.547+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v80: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:34 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:34.552+0000 7f6b98cb8700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.ceph-node-00.cephlab.com (monmap changed)...
May 27 13:14:34 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:34.556+0000 7f6b98cb8700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.ceph-node-00.cephlab.com on ceph-node-00.cephlab.com
May 27 13:14:35 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:35.395+0000 7f6b98cb8700 0 log_channel(cephadm) log [INF] : Reconfiguring prometheus.ceph-node-00 (dependencies changed)...
May 27 13:14:35 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:35.399+0000 7f6b98cb8700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon prometheus.ceph-node-00 on ceph-node-00.cephlab.com
May 27 13:14:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:36.547+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v81: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:37 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:37.256+0000 7f6b98cb8700 0 log_channel(cephadm) log [INF] : Reconfiguring alertmanager.ceph-node-00 (dependencies changed)...
May 27 13:14:37 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:37.259+0000 7f6b98cb8700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon alertmanager.ceph-node-00 on ceph-node-00.cephlab.com
May 27 13:14:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:38.548+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v82: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:39.183+0000 7f6b98cb8700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.ceph-node-00 (monmap changed)...
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:39.184+0000 7f6b98cb8700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'cephadm' while running on mgr.ceph-node-00.cephlab.com.mfduko: auth get failed: failed to find client.crash.ceph-node-00 in keyring
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:39.184+0000 7f6b98cb8700 -1 cephadm.serve:
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:39.184+0000 7f6b98cb8700 -1 Traceback (most recent call last):
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/module.py", line 502, in serve
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: self._check_daemons()
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1841, in _check_daemons
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: dd.hostname, reconfig=True)
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1568, in _create_daemon
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: extra_ceph_config=extra_config.pop('config', ''))
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1518, in _get_config_and_keyring
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: 'entity': ename,
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: File "/usr/share/ceph/mgr/mgr_module.py", line 1096, in check_mon_command
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: raise MonCommandFailed(f'{cmd_dict["prefix"]} failed: {r.stderr}')
May 27 13:14:39 ceph-node-00.cephlab.com bash[13093]: mgr_module.MonCommandFailed: auth get failed: failed to find client.crash.ceph-node-00 in keyring
May 27 13:14:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:40.548+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v83: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:42.549+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v84: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:44.550+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v85: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:46 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:46.550+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v86: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:47 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:14:47] "GET /metrics HTTP/1.1" 200 45365 "" "Prometheus/2.18.1"
May 27 13:14:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:48.550+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v87: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:48.994+0000 7f6ba1cca700 0 log_channel(audit) log [DBG] : from='client.14224 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch
May 27 13:14:50 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:50.551+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v88: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:52.551+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v89: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:54.551+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v90: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:56 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:56.552+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v91: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:14:57 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:14:57] "GET /metrics HTTP/1.1" 200 45372 "" "Prometheus/2.18.1"
May 27 13:14:58 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:14:58.552+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v92: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:00 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:00.553+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v93: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:02 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:02.554+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v94: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:04 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:04.554+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v95: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:06 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:06.555+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v96: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:07 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:15:07] "GET /metrics HTTP/1.1" 200 45376 "" "Prometheus/2.18.1"
May 27 13:15:08 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:08.555+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v97: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:10 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:10.556+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v98: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:12 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:12.556+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v99: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:14 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:14.556+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v100: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:16 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:16.557+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v101: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:17 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:15:17] "GET /metrics HTTP/1.1" 200 45377 "" "Prometheus/2.18.1"
May 27 13:15:18 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:18.557+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v102: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:20 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:20.558+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v103: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:22 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:22.558+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v104: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:24 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:24.559+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v105: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:26 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:26.559+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v106: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:27 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:15:27] "GET /metrics HTTP/1.1" 200 45376 "" "Prometheus/2.18.1"
May 27 13:15:28 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:28.559+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v107: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:30 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:30.560+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v108: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:32 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:32.561+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v109: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:34 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:34.561+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v110: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:36.561+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v111: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:37 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:15:37] "GET /metrics HTTP/1.1" 200 45377 "" "Prometheus/2.18.1"
May 27 13:15:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:38.562+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v112: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:40.562+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v113: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:42.563+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v114: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:44.563+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v115: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:46 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:46.563+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v116: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:47 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:15:47] "GET /metrics HTTP/1.1" 200 45378 "" "Prometheus/2.18.1"
May 27 13:15:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:48.564+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v117: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:50 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:50.564+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v118: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:52.565+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v119: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:54.565+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v120: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:56 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:56.566+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v121: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:15:57 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:15:57] "GET /metrics HTTP/1.1" 200 45385 "" "Prometheus/2.18.1"
May 27 13:15:58 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:15:58.567+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v122: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:00 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:00.567+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v123: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:02 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:02.568+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v124: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:04 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:04.568+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v125: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:06 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:06.568+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v126: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:07 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:16:07] "GET /metrics HTTP/1.1" 200 45388 "" "Prometheus/2.18.1"
May 27 13:16:08 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:08.569+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v127: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:10 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:10.569+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v128: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:12 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:12.570+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v129: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:14 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:14.570+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v130: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:16 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:16.571+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v131: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:17 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:16:17] "GET /metrics HTTP/1.1" 200 45390 "" "Prometheus/2.18.1"
May 27 13:16:18 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:18.571+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v132: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:20 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:20.571+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v133: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:22 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:22.572+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v134: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:24 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:24.572+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v135: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:26 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:26.573+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v136: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:27 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:16:27] "GET /metrics HTTP/1.1" 200 45388 "" "Prometheus/2.18.1"
May 27 13:16:28 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:28.573+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v137: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:30 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:30.573+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v138: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:32 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:32.574+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v139: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:34 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:34.574+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v140: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:36 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:36.575+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v141: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:37 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:16:37] "GET /metrics HTTP/1.1" 200 45389 "" "Prometheus/2.18.1"
May 27 13:16:38 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:38.576+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v142: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:40 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:40.576+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v143: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:42 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:42.576+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v144: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:44 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:44.576+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v145: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:46 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:46.578+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v146: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:47 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:16:47] "GET /metrics HTTP/1.1" 200 45388 "" "Prometheus/2.18.1"
May 27 13:16:48 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:48.578+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v147: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:50 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:50.579+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v148: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:52 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:52.579+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v149: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:54 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:54.580+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v150: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:56 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:56.580+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v151: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:16:57 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:16:57] "GET /metrics HTTP/1.1" 200 45388 "" "Prometheus/2.18.1"
May 27 13:16:58 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:16:58.580+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v152: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:00 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:00.581+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v153: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:02 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:02.581+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v154: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:04 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:04.582+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v155: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:06 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:06.582+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v156: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:07 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:17:07] "GET /metrics HTTP/1.1" 200 45390 "" "Prometheus/2.18.1"
May 27 13:17:08 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:08.583+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v157: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:10 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:10.583+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v158: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:12 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:12.583+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v159: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:14 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:14.584+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v160: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:16 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:16.584+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v161: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:17 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:17:17] "GET /metrics HTTP/1.1" 200 45385 "" "Prometheus/2.18.1"
May 27 13:17:18 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:18.584+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v162: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:20 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:20.585+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v163: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:22 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:22.585+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v164: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:24 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:24.585+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v165: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:26 ceph-node-00.cephlab.com bash[13093]: debug 2020-05-27T13:17:26.586+0000 7f6ba0cc8700 0 log_channel(cluster) log [DBG] : pgmap v166: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail
May 27 13:17:27 ceph-node-00.cephlab.com bash[13093]: ::ffff:192.168.122.194 - - [27/May/2020:13:17:27] "GET /metrics HTTP/1.1" 200 45390 "" "Prometheus/2.18.1"
    (1-1/1)