Project

General

Profile

Bug #52002

mgr/dashboard: dashboard 16.2.5 unable to ipv6 wildcard bind

Added by Scott Shambarger over 1 year ago. Updated 4 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Community (user)
Tags:
backport_processed
Backport:
pacific, octopus
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Description of problem

Commit 157a7b418 (cherry picked as ad964c42a) updated how the service URL was calculated, but appears to have (inadvertently?) changed the bind address as well.

Affected file: src/pybind/mgr/dashboard/module.py around line 110

If server_addr is "::", it replaces it using get_mgr_ip(). However, the modified variable is used for the CherryPy config option 'server.socket_host' which causes the Dashboard to only bind to the individual manager ip (and not the wildcard address as in previous versions, and as documented).

A workaround for ipv4 can be accomplished by setting server_addr to 0.0.0.0, but then the URL is incorrect. There is no longer any way to bind to the ipv6 wildcard...

The fix should be to use the get_mgr_ip for the URL specifically, and leave the server_addr alone.

Additionally, the URL should be fixed to use the manager ip if the server_addr is :: OR 0.0.0.0.

Environment

  • ceph version string: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
  • Platform (OS/distro/release): Fedora 34
  • Cluster details (nodes, monitors, OSDs): ...
  • Did it happen on a stable environment or after a migration/upgrade?: cephadm install
  • Browser used (e.g.: Version 86.0.4240.198 (Official Build) (64-bit)): Firefox 90

How reproducible

Steps:

  1. Using multi-homed host with multiple networks (eg 10.0.10.50/24, 192.168.10.50/24)
  2. Install using cephadm with dashboard defaults, with --mon-addrv "[v2:10.0.10.50:3300]"
  3. Attempt to load the dashboard on a network that differs from the public_network, eg https://192.168.10.50:8443

Actual results

Connection fails. Dashboard is bound to a single IP address:
$ ss -lnt | grep 8443
LISTEN 0 5 10.0.10.32:8443 0.0.0.0:*

Expected results

Dashboard should be available on all interfaces:
$ ss -lnt | grep 8443
LISTEN 0 128 0.0.0.0:8443 0.0.0.0:*
LISTEN 0 128 [::]:8443 [::]:*

Additional info

The initial commit also made the same change to the prometheus module

File: src/pybind/mgr/prometheus/module.py around line 1380

It also changed the wildcard address... and there is no workaround as it affects both ":" and "0.0.0.0" server_addr values.


Related issues

Copied to Dashboard - Backport #52802: octopus: mgr/dashboard: dashboard 16.2.5 unable to ipv6 wildcard bind New
Copied to Dashboard - Backport #52803: pacific: mgr/dashboard: dashboard 16.2.5 unable to ipv6 wildcard bind Resolved

History

#1 Updated by Ernesto Puerta over 1 year ago

  • Project changed from Dashboard to Orchestrator
  • Category deleted (Component - Services & Daemons)

Cephadm folks: what do you think about this one?

#2 Updated by Scott Shambarger over 1 year ago

Fix is pretty straightforward...submitted a PR https://github.com/ceph/ceph/pull/42716

#3 Updated by Sebastian Wagner over 1 year ago

  • Project changed from Orchestrator to Dashboard

#4 Updated by Avan Thakkar over 1 year ago

  • Status changed from New to Duplicate

#5 Updated by Avan Thakkar over 1 year ago

Duplicate of https://tracker.ceph.com/issues/52117. Already fixed.

#6 Updated by Alfonso Martínez over 1 year ago

  • Pull request ID set to 42716

#7 Updated by Scott Shambarger over 1 year ago

Avan Thakkar wrote:

Duplicate of https://tracker.ceph.com/issues/52117. Already fixed.

Not sure why this was closed as duplicate... issue 52117 is related to the dashboard URL, and this issue is on the address dashboard is binding to (as in it is not binding to the wildcard address).

Please re-open (I don't seem to be able to update the status...)

#8 Updated by Ernesto Puerta over 1 year ago

  • Status changed from Duplicate to New

#9 Updated by Daniël Vos about 1 year ago

I typed a big bug report to find out this bug report covers the same problem, just different symptoms

Perhaps my findings are of some use, I'll paste it here:

Description of problem

After upgrade Octopus -> Pacific I found my ceph-mgr processes listening on <node-ip>:<port> instead of *:<port>.

Ceph Mgr Dashboard Documentation states default is :: if mgr/dashboard/server_addr is left unconfigured but this is not the case.

https://docs.ceph.com/en/latest/mgr/dashboard/?highlight=backfill#:~:text=If%20no%20specific%20address%20has%20been%20configured%2C%20the%20web%20app%20will%20bind%20to%20%3A%3A%2C%20which%20corresponds%20to%20all%20available%20IPv4%20and%20IPv6%20addresses.

Environment

  • ceph version string: 16.2.5
  • Platform (OS/distro/release): Ubuntu 20.04
  • Cluster details (nodes, monitors, OSDs): 3 nodes, 3 mons, 3 mgrs, 4 cephfs, 20 OSDs
  • Did it happen on a stable environment or after a migration/upgrade?: Problem found after upgrade from Ceph 15.2.14
  • Browser used (e.g.: Version 86.0.4240.198 (Official Build) (64-bit)): N/A

How reproducible

Steps:

  1. Leave `mgr mgr/dashboard/server_addr` config default, documentation states the ceph-mgr listener will be ::

Actual results

ceph-mgr listener is <ip-address>:<port>

~
root @ node2 # ceph config get mgr mgr/dashboard/server_addr
::

~
root @ node2 # ceph config get mgr mgr/dashboard/server_port
7080

~
root @ node2 # ss -tlpn | grep ceph-mgr
LISTEN 0      512      [2001:db8:2000:8a:67:616e:6573:6861]:6800           [::]:*                      users:(("ceph-mgr",pid=2997269,fd=24))
LISTEN 0      512      [2001:db8:2000:8a:67:616e:6573:6861]:6801           [::]:*                      users:(("ceph-mgr",pid=2997269,fd=28))
LISTEN 0      5        [2001:db8:2000:8a:67:616e:6573:6861]:9283           [::]:*                      users:(("ceph-mgr",pid=2997269,fd=43))
LISTEN 0      5        [2001:db8:2000:8a:67:616e:6573:6861]:7080           [::]:*                      users:(("ceph-mgr",pid=2997269,fd=56))

Expected results

A listener on [::]:<server_port>, as is the default stated by the documentation

Additional info

I've tried configuring `mgr/dashboard/server_addr` with alternative addresses, like such

ceph config set mgr mgr/dashboard/server_addr '::'
ceph config set mgr mgr/dashboard/server_addr '[::]'
ceph config set mgr mgr/dashboard/server_addr '*'

The first doesn't do what it is supposed to do (This is also the default). It gives you a listener on <ip-address>:<port>

The second option makes the mgr crash at start-up, obviously invalid. (gaierror(-2, 'Name or service not known'))

The last also makes the mgr crash at start-up (OSError("No socket could be created -- (('0.0.0.0', 7080): [Errno 2] Name or service not known) - (('::', 7080, 0, 0): [Errno -2] Name or service not known)",))

My other Ceph Octopus cluster has `mgr/dashboard/server_addr` unconfigured and here all ceph-mgr processes are listening on *, but more specifically so is the ceph-mgr process for the dashboard. So it seems like a recent change.

root @ mon1.ceph2 # ss -tlpn | grep ceph-mgr
LISTEN   0        5                             *:7443                 *:*       users:(("ceph-mgr",pid=1818,fd=27))
LISTEN   0        5                             *:9283                 *:*       users:(("ceph-mgr",pid=1818,fd=24))

#10 Updated by Pere Díaz Bou about 1 year ago

  • Status changed from New to Fix Under Review

#11 Updated by Ernesto Puerta about 1 year ago

  • Status changed from Fix Under Review to Pending Backport

#12 Updated by Backport Bot about 1 year ago

  • Copied to Backport #52802: octopus: mgr/dashboard: dashboard 16.2.5 unable to ipv6 wildcard bind added

#13 Updated by Backport Bot about 1 year ago

  • Copied to Backport #52803: pacific: mgr/dashboard: dashboard 16.2.5 unable to ipv6 wildcard bind added

#14 Updated by Backport Bot 4 months ago

  • Tags set to backport_processed

Also available in: Atom PDF