Project

General

Profile

Actions

Bug #51446

closed

Module 'dashboard' has failed: Timeout('Port 8443 not bound on 192.168.122.121.',)

Added by Daniel Pivonka almost 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

standby dashboard is binding to 127.0.1.1 becasue get_mgr_ip() always returns hostname for standby mgr and with podmans /etc/host file the hostname resolves to 127.0.1.1 when failing over to the standby mgr it crashes cause cherry py is still binded to 127.0.1.1 when it expects it to be binded to the mgr ip


Related issues 2 (0 open2 closed)

Related to Dashboard - Bug #37651: Container, Mimic: Unable to open CherryPy portCan't reproduce

Actions
Related to Dashboard - Bug #38411: Under rook+minikube: HEALTH_ERR: Module 'dashboard' has failed: IOError("Port 8443 not bound on '::'",)Can't reproduce

Actions
Actions #1

Updated by Sage Weil almost 3 years ago

  • Project changed from Ceph to Orchestrator
Actions #2

Updated by Patrick Seidensal almost 3 years ago

This might not be a pure orchestrator issue. I've investigated the `run-backend-api-tests.sh` issue I have locally and the cause seems to be the same. As the change to determine the host name or IP address of the dashboard module is about a month old, I suspect that the upstream tests have not suffered by that change, but it might be the cause.

https://github.com/ceph/ceph/commit/157a7b4183dbd888f106c613a758409d7e07b917

Whatever might fix that incident for podman, it seems it'll do so for running the Teuthology tests locally, too.

 2021-06-30T16:52:11.364+0200 7f3da7c08700 -1 Traceback (most recent call last):
   File "/home/user/src/ceph/src/pybind/mgr/dashboard/module.py", line 343, in serve
     cherrypy.engine.start()
   File "/usr/lib/python3/dist-packages/cherrypy/process/wspbus.py", line 264, in start
     self.exit()
   File "/usr/lib/python3/dist-packages/cherrypy/process/wspbus.py", line 254, in start
     self.publish('start')
   File "/usr/lib/python3/dist-packages/cherrypy/process/wspbus.py", line 234, in publish
     raise exc
 cherrypy.process.wspbus.ChannelFailures: OSError("Port 8890 not bound on '192.168.1.140'")

LISTEN    0         5            192.168.1.140:8889             0.0.0.0:*        users:(("ceph-mgr",pid=66141,fd=40))                                           
LISTEN    0         5                127.0.1.1:8890             0.0.0.0:*        users:(("ceph-mgr",pid=66129,fd=24))                                           
LISTEN    0         5                127.0.1.1:8891             0.0.0.0:*        users:(("ceph-mgr",pid=66134,fd=24))                                           

It just works when I tell the dashboard module to listen on '0.0.0.0'.

(Note that I experimented with the ports and had them starting with 8889 instead of 7789 on my system currently)

Actions #3

Updated by Daniel Pivonka almost 3 years ago

  • Status changed from New to In Progress
  • Pull request ID set to 42107
Actions #4

Updated by Patrick Seidensal almost 3 years ago

  • Related to Bug #37651: Container, Mimic: Unable to open CherryPy port added
Actions #5

Updated by Patrick Seidensal almost 3 years ago

  • Related to Bug #38411: Under rook+minikube: HEALTH_ERR: Module 'dashboard' has failed: IOError("Port 8443 not bound on '::'",) added
Actions #6

Updated by Sage Weil almost 3 years ago

  • Status changed from In Progress to Resolved
Actions #7

Updated by Roland Sommer almost 3 years ago

Despite being closed, the change does affect certain setups using HA-IPs and/or IPVS because contrary to the docs saying that the dashboard will listen on every (local) IP if not configured otherwise, it now does not anymore. We fixed this using '0.0.0.0' for server_addr, but the docs should be adjusted if this is the new expected behaviour, because the following is just not correct anymore:

If no specific address has been configured, the web app will bind to ::, which corresponds to all available IPv4 and IPv6 addresses.

Actions #8

Updated by Roland Sommer almost 3 years ago

The same for all other places, where now get_mgr_ip is used. For example the prometheus module: docs say default is 0.0.0.0 - which is also not true anymore, breaking configurations relying on the any-listen-semantics.

Actions

Also available in: Atom PDF