Ceph » mgr

Hello !

I've seen the https://tracker.ceph.com/issues/63388 issue, and I'm currently facing something a bit similar.
I'm also on an IPv6-only cluster, if it matters, and running ceph 18.2.2 provisioned by cephadm.
For some reason, service_discovery service is not starting, causing prometheus to not have any targets, and I don't see any obvious reason why not, nor anything in the logs I can work with.

Here are some info :

$ ceph mgr services
{
    "dashboard": "https://[2a02:175a:...]:8443/",
    "prometheus": "http://[2a02:175a:...]:9283/" 
}

I have nothing on port 8765 despite cephadm being loaded

$ ceph mgr module ls
...
cephadm               on
...

Logs from a MGR after a reboot :

May 03 16:14:22 storage06 bash[709770]: debug 2024-05-03T14:14:22.886+0000 7fa7efdfe200  0 set uid:gid to 167:167 (ceph:ceph)
May 03 16:14:22 storage06 bash[709770]: debug 2024-05-03T14:14:22.886+0000 7fa7efdfe200  0 ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable), process ceph-mgr, >
May 03 16:14:22 storage06 bash[709770]: debug 2024-05-03T14:14:22.886+0000 7fa7efdfe200  0 pidfile_write: ignore empty --pid-file
May 03 16:14:22 storage06 bash[709770]: debug 2024-05-03T14:14:22.934+0000 7fa7efdfe200  1 mgr[py] Loading python module 'alerts'
May 03 16:14:23 storage06 bash[709770]: debug 2024-05-03T14:14:23.174+0000 7fa7efdfe200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member
May 03 16:14:23 storage06 bash[709770]: debug 2024-05-03T14:14:23.174+0000 7fa7efdfe200  1 mgr[py] Loading python module 'balancer'
May 03 16:14:23 storage06 bash[709770]: debug 2024-05-03T14:14:23.370+0000 7fa7efdfe200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member
May 03 16:14:23 storage06 bash[709770]: debug 2024-05-03T14:14:23.370+0000 7fa7efdfe200  1 mgr[py] Loading python module 'cephadm'
May 03 16:14:24 storage06 bash[709770]: debug 2024-05-03T14:14:24.730+0000 7fa7efdfe200  1 mgr[py] Loading python module 'crash'
May 03 16:14:24 storage06 bash[709770]: debug 2024-05-03T14:14:24.946+0000 7fa7efdfe200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member
May 03 16:14:24 storage06 bash[709770]: debug 2024-05-03T14:14:24.946+0000 7fa7efdfe200  1 mgr[py] Loading python module 'dashboard'
May 03 16:14:26 storage06 bash[709770]: debug 2024-05-03T14:14:26.230+0000 7fa7efdfe200  1 mgr[py] Loading python module 'devicehealth'
May 03 16:14:26 storage06 bash[709770]: debug 2024-05-03T14:14:26.570+0000 7fa7efdfe200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member
May 03 16:14:26 storage06 bash[709770]: debug 2024-05-03T14:14:26.570+0000 7fa7efdfe200  1 mgr[py] Loading python module 'diskprediction_local'
May 03 16:14:26 storage06 bash[709770]: debug 2024-05-03T14:14:26.974+0000 7fa7efdfe200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member
May 03 16:14:26 storage06 bash[709770]: debug 2024-05-03T14:14:26.974+0000 7fa7efdfe200  1 mgr[py] Loading python module 'influx'
May 03 16:14:27 storage06 bash[709770]: debug 2024-05-03T14:14:27.194+0000 7fa7efdfe200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member
May 03 16:14:27 storage06 bash[709770]: debug 2024-05-03T14:14:27.194+0000 7fa7efdfe200  1 mgr[py] Loading python module 'insights'
May 03 16:14:27 storage06 bash[709770]: debug 2024-05-03T14:14:27.562+0000 7fa7efdfe200  1 mgr[py] Loading python module 'iostat'
May 03 16:14:27 storage06 bash[709770]: debug 2024-05-03T14:14:27.774+0000 7fa7efdfe200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member
May 03 16:14:27 storage06 bash[709770]: debug 2024-05-03T14:14:27.774+0000 7fa7efdfe200  1 mgr[py] Loading python module 'k8sevents'
May 03 16:14:28 storage06 bash[709770]: debug 2024-05-03T14:14:28.886+0000 7fa7efdfe200  1 mgr[py] Loading python module 'localpool'
May 03 16:14:29 storage06 bash[709770]: debug 2024-05-03T14:14:29.122+0000 7fa7efdfe200  1 mgr[py] Loading python module 'mds_autoscaler'
May 03 16:14:29 storage06 bash[709770]: debug 2024-05-03T14:14:29.562+0000 7fa7efdfe200  1 mgr[py] Loading python module 'mirroring'
May 03 16:14:29 storage06 bash[709770]: debug 2024-05-03T14:14:29.830+0000 7fa7efdfe200  1 mgr[py] Loading python module 'nfs'
May 03 16:14:30 storage06 bash[709770]: debug 2024-05-03T14:14:30.578+0000 7fa7efdfe200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member
May 03 16:14:30 storage06 bash[709770]: debug 2024-05-03T14:14:30.578+0000 7fa7efdfe200  1 mgr[py] Loading python module 'orchestrator'
May 03 16:14:31 storage06 bash[709770]: debug 2024-05-03T14:14:31.074+0000 7fa7efdfe200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member
May 03 16:14:31 storage06 bash[709770]: debug 2024-05-03T14:14:31.074+0000 7fa7efdfe200  1 mgr[py] Loading python module 'osd_perf_query'
May 03 16:14:31 storage06 bash[709770]: debug 2024-05-03T14:14:31.334+0000 7fa7efdfe200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member
May 03 16:14:31 storage06 bash[709770]: debug 2024-05-03T14:14:31.334+0000 7fa7efdfe200  1 mgr[py] Loading python module 'osd_support'
May 03 16:14:31 storage06 bash[709770]: debug 2024-05-03T14:14:31.566+0000 7fa7efdfe200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member
May 03 16:14:31 storage06 bash[709770]: debug 2024-05-03T14:14:31.566+0000 7fa7efdfe200  1 mgr[py] Loading python module 'pg_autoscaler'
May 03 16:14:31 storage06 bash[709770]: debug 2024-05-03T14:14:31.842+0000 7fa7efdfe200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member
May 03 16:14:31 storage06 bash[709770]: debug 2024-05-03T14:14:31.842+0000 7fa7efdfe200  1 mgr[py] Loading python module 'progress'
May 03 16:14:32 storage06 bash[709770]: debug 2024-05-03T14:14:32.078+0000 7fa7efdfe200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
May 03 16:14:32 storage06 bash[709770]: debug 2024-05-03T14:14:32.078+0000 7fa7efdfe200  1 mgr[py] Loading python module 'prometheus'
May 03 16:14:33 storage06 bash[709770]: debug 2024-05-03T14:14:33.382+0000 7fa7efdfe200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member
May 03 16:14:33 storage06 bash[709770]: debug 2024-05-03T14:14:33.382+0000 7fa7efdfe200  1 mgr[py] Loading python module 'rbd_support'
May 03 16:14:33 storage06 bash[709770]: debug 2024-05-03T14:14:33.682+0000 7fa7efdfe200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member
May 03 16:14:33 storage06 bash[709770]: debug 2024-05-03T14:14:33.682+0000 7fa7efdfe200  1 mgr[py] Loading python module 'restful'
May 03 16:14:34 storage06 bash[709770]: debug 2024-05-03T14:14:34.242+0000 7fa7efdfe200  1 mgr[py] Loading python module 'rgw'
May 03 16:14:34 storage06 bash[709770]: debug 2024-05-03T14:14:34.750+0000 7fa7efdfe200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member
May 03 16:14:34 storage06 bash[709770]: debug 2024-05-03T14:14:34.750+0000 7fa7efdfe200  1 mgr[py] Loading python module 'rook'
May 03 16:14:36 storage06 bash[709770]: debug 2024-05-03T14:14:36.574+0000 7fa7efdfe200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member
May 03 16:14:36 storage06 bash[709770]: debug 2024-05-03T14:14:36.574+0000 7fa7efdfe200  1 mgr[py] Loading python module 'selftest'
May 03 16:14:36 storage06 bash[709770]: debug 2024-05-03T14:14:36.822+0000 7fa7efdfe200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member
May 03 16:14:36 storage06 bash[709770]: debug 2024-05-03T14:14:36.822+0000 7fa7efdfe200  1 mgr[py] Loading python module 'snap_schedule'
May 03 16:14:37 storage06 bash[709770]: debug 2024-05-03T14:14:37.090+0000 7fa7efdfe200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member
May 03 16:14:37 storage06 bash[709770]: debug 2024-05-03T14:14:37.090+0000 7fa7efdfe200  1 mgr[py] Loading python module 'stats'
May 03 16:14:37 storage06 bash[709770]: debug 2024-05-03T14:14:37.362+0000 7fa7efdfe200  1 mgr[py] Loading python module 'status'
May 03 16:14:37 storage06 bash[709770]: debug 2024-05-03T14:14:37.626+0000 7fa7efdfe200 -1 mgr[py] Module status has missing NOTIFY_TYPES member
May 03 16:14:37 storage06 bash[709770]: debug 2024-05-03T14:14:37.626+0000 7fa7efdfe200  1 mgr[py] Loading python module 'telegraf'
May 03 16:14:37 storage06 bash[709770]: debug 2024-05-03T14:14:37.926+0000 7fa7efdfe200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member
May 03 16:14:37 storage06 bash[709770]: debug 2024-05-03T14:14:37.926+0000 7fa7efdfe200  1 mgr[py] Loading python module 'telemetry'
May 03 16:14:38 storage06 bash[709770]: debug 2024-05-03T14:14:38.822+0000 7fa7efdfe200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member
May 03 16:14:38 storage06 bash[709770]: debug 2024-05-03T14:14:38.822+0000 7fa7efdfe200  1 mgr[py] Loading python module 'test_orchestrator'
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.230+0000 7fa7efdfe200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.230+0000 7fa7efdfe200  1 mgr[py] Loading python module 'volumes'
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.726+0000 7fa7efdfe200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.726+0000 7fa7efdfe200  1 mgr[py] Loading python module 'zabbix'
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.930+0000 7fa7efdfe200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.938+0000 7fa7e3c9a700  0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO (2/5)
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.938+0000 7fa7e3c9a700  1 mgr load Constructed class from module: dashboard
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.942+0000 7fa7e3499700  0 ms_deliver_dispatch: unhandled message 0x563261218f20 mon_map magic: 0 v1 from mon.3 v2:[2a02:17>
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.942+0000 7fa7e3c9a700  0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO (2/5)
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.942+0000 7fa7e3c9a700  1 mgr load Constructed class from module: prometheus
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.942+0000 7fa5185da700  0 [dashboard INFO root] server: ssl=yes host=:: port=8443
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.942+0000 7fa518ddb700  0 [prometheus INFO root] server_addr: :: server_port: 9283
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.942+0000 7fa518ddb700  0 [prometheus INFO root] Starting engine...
May 03 16:14:39 storage06 bash[709770]: [03/May/2024:14:14:39] ENGINE Bus STARTING
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.942+0000 7fa518ddb700  0 [prometheus INFO cherrypy.error] [03/May/2024:14:14:39] ENGINE Bus STARTING
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.990+0000 7fa5185da700  0 [dashboard INFO root] Configured CherryPy, starting engine...
May 03 16:14:39 storage06 bash[709770]: debug 2024-05-03T14:14:39.990+0000 7fa5185da700  0 [dashboard INFO root] Starting engine...
May 03 16:14:40 storage06 bash[709770]: CherryPy Checker:
May 03 16:14:40 storage06 bash[709770]: The Application mounted at '' has an empty config.
May 03 16:14:40 storage06 bash[709770]: [03/May/2024:14:14:40] ENGINE Serving on http://:::9283
May 03 16:14:40 storage06 bash[709770]: debug 2024-05-03T14:14:40.054+0000 7fa518ddb700  0 [prometheus INFO cherrypy.error] [03/May/2024:14:14:40] ENGINE Serving on http://:::9283
May 03 16:14:40 storage06 bash[709770]: [03/May/2024:14:14:40] ENGINE Bus STARTED
May 03 16:14:40 storage06 bash[709770]: debug 2024-05-03T14:14:40.054+0000 7fa518ddb700  0 [prometheus INFO cherrypy.error] [03/May/2024:14:14:40] ENGINE Bus STARTED
May 03 16:14:40 storage06 bash[709770]: debug 2024-05-03T14:14:40.054+0000 7fa518ddb700  0 [prometheus INFO root] Engine started.
May 03 16:14:40 storage06 bash[709770]: debug 2024-05-03T14:14:40.110+0000 7fa5185da700  0 [dashboard INFO root] Engine started...
May 03 16:29:39 storage06 bash[709770]: debug 2024-05-03T14:29:39.952+0000 7fa7e3499700  0 client.0 ms_handle_reset on v2:[2a02:175a:...]:6802/1951634768