Project

General

Profile

Bug #38411

Under rook+minikube: HEALTH_ERR: Module 'dashboard' has failed: IOError("Port 8443 not bound on '::'",)

Added by Jeff Layton about 1 month ago. Updated 23 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
dashboard/general
Target version:
-
Start date:
02/20/2019
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

TI have a minikube VM running (v0.34.1), and attempted to deploy a bootstrap rook/ceph cluster on it (starting only a single mon and a single mgr), and a toolbox container. I'm using nautilus images and a fairly bleeding-edge rook image. Once it all came up, I exec'ed a shell in the toolbox and did a ceph status:

# ceph status
  cluster:
    id:     065b0695-9293-4153-831e-a11919a4fe96
    health: HEALTH_ERR
            Module 'dashboard' has failed: IOError("Port 8443 not bound on '::'",)

  services:
    mon: 1 daemons, quorum a (age 40m)
    mgr: a(active, since 8s)
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     

Looking at the mgr logs, I see some backtraces in cherrypy code:

debug 2019-02-20 18:10:33.391 7fb4e4f07700  0 log_channel(cluster) log [DBG] : pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail
debug 2019-02-20 18:10:35.392 7fb4e4f07700  0 log_channel(cluster) log [DBG] : pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail
[20/Feb/2019:18:10:35] ENGINE Error in 'start' listener <bound method Server.start of <cherrypy._cpserver.Server object at 0x7fb4ed547ed0>>
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 197, in publish
    output.append(listener(*args, **kwargs))
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpserver.py", line 151, in start
    ServerAdapter.start(self)
  File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 174, in start
    self.wait()
  File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 214, in wait
    wait_for_occupied_port(host, port)
  File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 427, in wait_for_occupied_port
    raise IOError("Port %r not bound on %r" % (port, host))
IOError: Port 8443 not bound on '::'

[20/Feb/2019:18:10:35] ENGINE Shutting down due to error in start listener:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 235, in start
    self.publish('start')
  File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 215, in publish
    raise exc
ChannelFailures: IOError("Port 8443 not bound on '::'",)

[20/Feb/2019:18:10:35] ENGINE Bus STOPPING
[20/Feb/2019:18:10:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 8443)) already shut down
[20/Feb/2019:18:10:35] ENGINE Stopped thread '_TimeoutMonitor'.
[20/Feb/2019:18:10:35] ENGINE Bus STOPPED
[20/Feb/2019:18:10:35] ENGINE Bus EXITING
[20/Feb/2019:18:10:35] ENGINE Bus EXITED
debug 2019-02-20 18:10:35.833 7fb4e3704700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'dashboard' while running on mgr.a: IOError("Port 8443 not bound on '::'",)
debug 2019-02-20 18:10:35.833 7fb4e3704700 -1 dashboard.serve:
debug 2019-02-20 18:10:35.833 7fb4e3704700 -1 Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/module.py", line 327, in serve
    cherrypy.engine.start()
  File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 250, in start
    raise e_info
ChannelFailures: IOError("Port 8443 not bound on '::'",)

debug 2019-02-20 18:10:37.392 7fb4e4f07700  0 log_channel(cluster) log [DBG] : pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail
debug 2019-02-20 18:10:39.393 7fb4e4f07700  0 log_channel(cluster) log [DBG] : pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail

This all works on my other single-node, non-minikube cluster, so it may be due to some quirk of the networking under minikube. Still, the dashboard should handle such errors more gracefully in that case.


Related issues

Duplicates mgr - Bug #37651: Container, Mimic: Unable to open CherryPy port Can't reproduce 02/20/2019

History

#2 Updated by Jeff Layton 30 days ago

I'm not sure how to interpret those instructions for nautilus...

I can't really enable ipv6 under minikube. The way it works is not really set up for network customization and it doesn't configure IPv6 addrs (aside from the link-local address).

Here's my config dump:

# ceph config dump
WHO       MASK LEVEL    OPTION                            VALUE      RO 
  mgr          advanced mgr/orchestrator_cli/orchestrator rook       *  
    mgr.       advanced mgr/dashboard/server_port         8443       *  
    mgr.a      advanced mgr/dashboard/server_addr         172.17.0.7 *  
    mgr.a      advanced mgr/prometheus/server_addr        172.17.0.7 *  

It seems like the dashboard needs to be able to handle a v4-only environments, but maybe I'm misunderstanding how these settings should work.

#3 Updated by Jeff Layton 30 days ago

I did try adding a mgr. server_addr config-key with '::' in it and restarting the mgr, but it still didn't work. New config dump:

# ceph config dump
WHO       MASK LEVEL    OPTION                            VALUE      RO 
  mgr          advanced mgr/orchestrator_cli/orchestrator rook       *  
    mgr.       advanced mgr/dashboard/server_addr         ::         *  
    mgr.       advanced mgr/dashboard/server_port         8443       *  
    mgr.a      advanced mgr/dashboard/server_addr         172.17.0.7 *  
    mgr.a      advanced mgr/prometheus/server_addr        172.17.0.7 *  

#4 Updated by Jeff Layton 24 days ago

  • Status changed from New to Duplicate

#5 Updated by Jeff Layton 24 days ago

  • Parent task set to #37651

#6 Updated by Jeff Layton 24 days ago

  • Parent task deleted (#37651)

#7 Updated by Jeff Layton 24 days ago

  • Duplicates Bug #37651: Container, Mimic: Unable to open CherryPy port added

#8 Updated by Sebastian Wagner 23 days ago

  • Status changed from Duplicate to New

Also available in: Atom PDF