Ceph » mgr » Dashboard

From the cherrypy docs in https://docs.cherrypy.org/en/latest/deploy.html#reverse-proxying we will need to provide a way to configure the ceph-dashboard in proxy mode and specify the frontend base URL, so that cherrypy can do his work.

I'd like to add a bit of additional information to this.

One way of working around this issue is described here: https://blog.widodh.nl/2019/01/haproxy-in-front-of-ceph-manager-dashboard/

However as far as I can tell, this requires that

• HAProxy operates with mode http (which, in turn, requires SSL to be disabled on the backend),
• HAProxy is itself capable of resolving the Location received in the initial 303 response.

I can't think of a way this could be made to work if SSL terminates on the backend nodes, and HAProxy is configured with mode tcp.

Can you please rephrase the requirements that are needed by the Dashboard to work correctly with HAProxy. Please don't assume that every developer knows about this piece of software and how it works.

I should add one more thing here: even with SSL enabled on the backend, it would be possible to configure HAproxy with mode tcp and a custom health check. That way, HAproxy would still use simple TCP forwarding (which it needs to, in order to not break end-to-end encryption), but not only rely on the successful establishment of the connection for the up/down decision. Instead it would base that decision on whatever the custom health check returns.

Edit: this approach is hardly practical because HAProxy typically runs in a chroot. For such a custom check (a combination of option external-check and external-check command) to work, it would have to be a statically linked binary in the HAProxy chroot.

But that's a fairly convoluted solution for something that most unsuspecting users would expect to be a trivial problem. Also, that solution would always be specific to HAproxy, and would have to be replicated for nginx, F5, or whatever the popular loadbalancer du jour might be.

Usual caveat, by the way: I may be missing something blindingly obvious that solves this issue much more easily. In that case, I'll be more than happy to stand corrected.

One more option I'm currently looking into: combining mode tcp with option httpcheck. That might be reasonably simple to do. If I can come up with a reference configuration, I'll hack up a doc patch.

Edit: this approach doesn't work. option httpcheck apparently always uses unencrypted HTTP, so you can't use that against something that speaks HTTPS. All that HAProxy gets is HTTP 400s (Bad Request), and it considers all backends down.

Edit edit: option httpcheck can be made to use HTTPS, if a backend server is configured with the ssl option. (see note 13, below)

Florian Haas wrote:

I'm not sure what the "right" way to resolve this would be, to be honest. Perhaps include a configuration option to toggle the dashboard behavior on non-active mgrs?

Maybe this is right solution. I don't see a very strong reason to have the non-active mgrs listening for HTTP connections since we have an easy way to know the current working dashboard URL by doing:

# ceph mgr services | jq .dashboard

Just to be clear, if we don't accept TCP connections to the dashboard webserver port in non-active mgrs that will make HAProxy checks correctly tagging which instances are up or down, and SSL can be enabled in dashboard. Is that correct Florian?

Just to be clear, if we don't accept TCP connections to the dashboard webserver port in non-active mgrs that will make HAProxy checks correctly tagging which instances are up or down, and SSL can be enabled in dashboard. Is that correct Florian?

By my understanding, yes. If the dashboard just doesn't accept connections on non-active mgrs, then that should work on reverse-proxy configurations without SSL, with SSL terminated on the backend, and with SSL terminated on the proxy.

With much kludging, it does seem that this can be done after all.

Here's a backend configuration that does seem to do the trick, i.e. not intercept SSL traffic but select the correct (SSL-only) backend based on its HTTP status:

backend dashboard_back
  mode tcp
  option httpchk GET /
  http-check expect status 200
  server daisy 192.168.122.114:8443 ssl check
  server eric 192.168.122.115:8443 ssl check
  server frank 192.168.122.116:8443 ssl check

HAProxy also needs to know about what CA signs your certs. If you're using a self-signed cert on the backends, that'll only work with ssl verify none check.

Still doesn't work without a frontend that has a cert of its own. So this still counts as an intercept and isn't true e2e.

Run-down from comment in note 8 stands. If you want to proxy, you have to terminate SSL on the proxy. If if the dashboard is meant to support proper SSL that is terminated on the backend, I can't think of a way to make that work without ditching the redirect, and do as Ricardo says in note 12.

Sorry for all the noise, been down a half-dozen dead ends here.

Florian Haas wrote:

Just to be clear, if we don't accept TCP connections to the dashboard webserver port in non-active mgrs that will make HAProxy checks correctly tagging which instances are up or down, and SSL can be enabled in dashboard. Is that correct Florian?

By my understanding, yes. If the dashboard just doesn't accept connections on non-active mgrs, then that should work on reverse-proxy configurations without SSL, with SSL terminated on the backend, and with SSL terminated on the proxy.

So it seems we have three options on how to resolve this:

1. Make it possible to enable/disable the HTTP 303 redirection on standby ceph-mgr instances, by disabling the dashboard's standby mode, so a HTTP connection to the dashboard's TCP port on a standby manager instance results in a "Connection refused error".
2. Allow customizing the URL that is returned by the HTTP 303 redirect message on standby instances so that it points to the proxy's public host name or IP address instead of the URL of the active ceph-mgr instance that currently serves the dashboard
3. Write a custom health check for HAproxy as outlined in https://www.loadbalancer.org/blog/how-to-write-an-external-custom-healthcheck-for-haproxy/ that ?

Does that summarize the possible options correctly?
Any suggestions/preferences on how to address this shortcoming?
To me, 1 or 2 sounds like the most feasible options, but I wonder if we need to implement both or if one of them would be sufficient? Option 3 would be very specific to HAProxy.

Lenz Grimmer wrote:

Does that summarize the possible options correctly?

From my point of view, yes it does.

Any suggestions/preferences on how to address this shortcoming?
To me, 1 or 2 sounds like the most feasible options, but I wonder if we need to implement both or if one of them would be sufficient? Option 3 would be very specific to HAProxy.

Agree with that assessment as well. I'd say that the option 1 is preferable. That gives users the option of terminating HTTPS either on the load balancer or on the mgr node, as applicable policy (or preference) dictates.

What do others think?

Relates to https://github.com/rook/rook/pull/3076#issuecomment-488012764

Does this also affect Prometheus or mgr/restful? If yes, we should update the ticket to reflect this to be a general mgr issue

Dashboard through proxy doesn't play nicely with grafana

When browser asks https://<proxyserver>/api/grafana/url the response is

{"instance": "http://grafana-host:3000/"}

which the client, in my case, doesn't know how to resolve or reach

One more thing, in relation to my previous comment:

when proxy is using TLS, the browser doesn't like that the dashboard tries to include non-tls http elements from grafana.

Torben Hørup wrote:

when proxy is using TLS, the browser doesn't like that the dashboard tries to include non-tls http elements from grafana.

Indeed, that's why we added the following note to the dashboard documentation

Ceph Dashboard embeds the Grafana dashboards via iframe HTML elements. If Grafana is configured without SSL/TLS support, most browsers will block the embedding of insecure content into a secured web page, if the SSL support in the dashboard has been enabled (which is the default configuration). If you can’t see the embedded Grafana dashboards after enabling them as outlined above, check your browser’s documentation on how to unblock mixed content. Alternatively, consider enabling SSL/TLS support in Grafana.

I've added a HAProxy setup for our development environment, see https://github.com/ricardoasmarques/ceph-dev-docker/pull/75/files.

I was able to get it running, but still get redirected if a failover occurs between two HAProxy health checks. In this case HAProxy has not realized that the node marked as active is now down. The node which is now down sends the redirection to the frontend client. Because HAProxy acts in SSL pass through mode it can not modify the headers. So the only workaround for this will be to add an option in Ceph Dashboard to prevent the redirection to the active node.

Torben Hørup wrote:

Dashboard through proxy doesn't play nicely with grafana

When browser asks https://<proxyserver>/api/grafana/url the response is

[...]

which the client, in my case, doesn't know how to resolve or reach

The Grafana Host must be reachable from outside (HAProxy frontend) to get working.