Ceph » mgr » Dashboard

So, my mgr is completely messed up. It crashes every few hours due to an ssl issue. Furthermore it is frozen, trying to get status for PG balancer or autoscaler yields in a hanging request. Restarting and having mgr failover to standby helps for a few minutes but then the same happens there.

2020-02-23 14:27:08.847 7f92ba61e700 -1 mgr handle_mgr_map I was active but no longer am
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn  e: '/usr/bin/ceph-mgr'
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn  0: '/usr/bin/ceph-mgr'
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn  1: '-f'
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn  2: '--cluster'
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn  3: 'ceph'
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn  4: '--id'
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn  5: 'm1-1045558'
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn  6: '--setuser'
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn  7: 'ceph'
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn  8: '--setgroup'
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn  9: 'ceph'
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn respawning with exe /usr/bin/ceph-mgr
2020-02-23 14:27:08.847 7f92ba61e700  1 mgr respawn  exe_path /proc/self/exe
2020-02-23 14:27:08.919 7f9ca84d1d40  0 ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable), process ceph-mgr, pid 2033058
2020-02-23 14:27:08.919 7f9ca84d1d40  0 pidfile_write: ignore empty --pid-file
2020-02-23 14:27:08.935 7f9ca84d1d40  1 mgr[py] Loading python module 'ansible'
2020-02-23 14:27:09.067 7f9ca84d1d40  1 mgr[py] Loading python module 'balancer'
2020-02-23 14:27:09.087 7f9ca84d1d40  1 mgr[py] Loading python module 'crash'
2020-02-23 14:27:09.103 7f9ca84d1d40  1 mgr[py] Loading python module 'dashboard'
2020-02-23 14:27:09.411 7f9ca84d1d40  1 mgr[py] Loading python module 'deepsea'
2020-02-23 14:27:09.539 7f9ca84d1d40  1 mgr[py] Loading python module 'devicehealth'
2020-02-23 14:27:09.555 7f9ca84d1d40  1 mgr[py] Loading python module 'diskprediction_cloud'
2020-02-23 14:27:09.591 7f9ca84d1d40  1 mgr[py] Loading python module 'diskprediction_local'
2020-02-23 14:27:09.611 7f9ca84d1d40  1 mgr[py] Loading python module 'influx'
2020-02-23 14:27:09.663 7f9ca84d1d40  1 mgr[py] Loading python module 'insights'
2020-02-23 14:27:09.679 7f9ca84d1d40  1 mgr[py] Loading python module 'iostat'
2020-02-23 14:27:09.695 7f9ca84d1d40  1 mgr[py] Loading python module 'localpool'
2020-02-23 14:27:09.707 7f9ca84d1d40  1 mgr[py] Loading python module 'orchestrator_cli'
2020-02-23 14:27:09.743 7f9ca84d1d40  1 mgr[py] Loading python module 'pg_autoscaler'
2020-02-23 14:27:09.783 7f9ca84d1d40  1 mgr[py] Loading python module 'progress'
2020-02-23 14:27:09.811 7f9ca84d1d40  1 mgr[py] Loading python module 'prometheus'
2020-02-23 14:27:09.959 7f9ca84d1d40  1 mgr[py] Loading python module 'rbd_support'
2020-02-23 14:27:10.047 7f9ca84d1d40  1 mgr[py] Loading python module 'restful'
2020-02-23 14:27:10.187 7f9ca84d1d40  1 mgr[py] Loading python module 'rook'
2020-02-23 14:27:10.223 7f9ca84d1d40  1 mgr[py] Loading python module 'selftest'
2020-02-23 14:27:10.239 7f9ca84d1d40  1 mgr[py] Loading python module 'ssh'
2020-02-23 14:27:10.279 7f9ca84d1d40  1 mgr[py] Loading python module 'status'
2020-02-23 14:27:10.303 7f9ca84d1d40  1 mgr[py] Loading python module 'telegraf'
2020-02-23 14:27:10.323 7f9ca84d1d40  1 mgr[py] Loading python module 'telemetry'
2020-02-23 14:27:10.527 7f9ca84d1d40  1 mgr[py] Loading python module 'test_orchestrator'
2020-02-23 14:27:10.563 7f9ca84d1d40  1 mgr[py] Loading python module 'volumes'
2020-02-23 14:27:10.607 7f9ca84d1d40  1 mgr[py] Loading python module 'zabbix'
2020-02-23 14:27:10.643 7f9c950a0700  1 mgr load Constructed class from module: dashboard
2020-02-23 14:27:10.643 7f9c950a0700  1 mgr load Constructed class from module: prometheus
2020-02-23 14:27:10.643 7f9c9489f700  0 ms_deliver_dispatch: unhandled message 0x55a3519d3a00 mon_map magic: 0 v1 from mon.0 v2:10.3.2.1:3300/0
2020-02-23 14:27:10.643 7f9c9489f700  0 client.0 ms_handle_reset on v2:10.3.2.2:6802/3930062
2020-02-23 14:27:10.883 7f9c7f545700  0 mgr[dashboard] [23/Feb/2020:14:27:10] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start
    self.tick()
  File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick
    s, ssl_env = self.ssl_adapter.wrap(s)
  File "/usr/lib/python2.7/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 67, in wrap
    server_side=True)
  File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 617, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 846, in do_handshake
    self._sslobj.do_handshake()
error: [Errno 0] Error