Feature #64318
openmgr/prometheus add support for TLS and client cert authentication
0%
Description
Support for using TLS in all sorts of ways has come to a lot of (written in Golang) exporters via the exporter-toolkit (https://github.com/prometheus/exporter-toolkit/blob/master/CHANGELOG.md) or via other ways.
Looking at https://docs.ceph.com/en/latest/mgr/prometheus/#configuration, the Prometheus module of the mgr does not have any support for using TLS.
Be it for the HTTP server itself or to authenticate clients with client certificates. Both of which Prometheus can quite easily make use of by setting tls_config (https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config) within the scrape config.
It would be good to be able to protect Ceph metrics endpoints via TLS transport encryption and also client certificates.
The same actually also applies to the ceph-exporter (https://github.com/ceph/ceph/tree/main/src/exporter).
There is an (unofficial) client library for C++ at https://github.com/jupp0r/prometheus-cpp, I don't know this makes sense to use for Ceph.
Updated by Redouane Kachach Elhichou 2 months ago
cephadm has already support to enable security across all the monitoring stack (including all the components). The configuration variable is mgr/cephadm/secure_monitoring_stack maybe it's just a documentation effort in this case.
Updated by Christian Rohmann 2 months ago
Redouane Kachach Elhichou wrote:
cephadm has already support to enable security across all the monitoring stack (including all the components). The configuration variable is mgr/cephadm/secure_monitoring_stack maybe it's just a documentation effort in this case.
I am not sure really. What does this setting actually do for the various components?
Looking at https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/services/monitoring.py it appears it's using https to reach the service_discovery of the mgr or adds authentication to Prometheus + Alertmanager.
But is there any TLS added? Looking at https://docs.ceph.com/en/reef/mgr/dashboard/#dashboard-ssl-tls-support I see that the dashboards does support TLS certificates. But the Prometheus module apparently does not have this capability: https://docs.ceph.com/en/reef/mgr/prometheus/#prometheus-module
Also the ceph-exporter distributed to Ceph node and exposes metrics for the local Ceph daemons. But it seems there is no authentication or TLS encryption support if you look at the code and the configuration options: https://github.com/ceph/ceph/blob/main/src/exporter/ceph_exporter.cc
Updated by Redouane Kachach Elhichou 2 months ago
Christian Rohmann wrote:
Redouane Kachach Elhichou wrote:
cephadm has already support to enable security across all the monitoring stack (including all the components). The configuration variable is mgr/cephadm/secure_monitoring_stack maybe it's just a documentation effort in this case.
I am not sure really. What does this setting actually do for the various components?
It enables security (SSL/TLS and basic auth) for all the monitoring components.
Looking at https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/services/monitoring.py it appears it's using https to reach the service_discovery of the mgr or adds authentication to Prometheus + Alertmanager.
But is there any TLS added? Looking at https://docs.ceph.com/en/reef/mgr/dashboard/#dashboard-ssl-tls-support I see that the dashboards does support TLS certificates. But the Prometheus module apparently does not have this capability: https://docs.ceph.com/en/reef/mgr/prometheus/#prometheus-module
That's right. Docs need to be updated to reflect how to enable security and how the components are impacted.
Also the ceph-exporter distributed to Ceph node and exposes metrics for the local Ceph daemons. But it seems there is no authentication or TLS encryption support if you look at the code and the configuration options: https://github.com/ceph/ceph/blob/main/src/exporter/ceph_exporter.cc
That's a known issue. In case of small clusters you can disable ceph-exporter by using exclude_perf_counters configuration parameter: https://docs.ceph.com/en/latest/mgr/prometheus/#confval-mgr-prometheus-exclude_perf_counters
Updated by Christian Rohmann 2 months ago
Redouane Kachach Elhichou wrote:
But is there any TLS added? Looking at https://docs.ceph.com/en/reef/mgr/dashboard/#dashboard-ssl-tls-support I see that the dashboards does support TLS certificates. But the Prometheus module apparently does not have this capability: https://docs.ceph.com/en/reef/mgr/prometheus/#prometheus-module
That's right. Docs need to be updated to reflect how to enable security and how the components are impacted.
So the prometheus mgr module actually does support TLS? Just to serve or also for client cert auth?
Is there any issue tracking these missing docs?
Also the ceph-exporter distributed to Ceph node and exposes metrics for the local Ceph daemons. But it seems there is no authentication or TLS encryption support if you look at the code and the configuration options: https://github.com/ceph/ceph/blob/main/src/exporter/ceph_exporter.cc
That's a known issue. In case of small clusters you can disable ceph-exporter by using exclude_perf_counters configuration parameter: https://docs.ceph.com/en/latest/mgr/prometheus/#confval-mgr-prometheus-exclude_perf_counters
If this is a known issue, is there an issue or Trello card to track?
Updated by Redouane Kachach Elhichou 2 months ago
Christian Rohmann wrote:
Redouane Kachach Elhichou wrote:
But is there any TLS added? Looking at https://docs.ceph.com/en/reef/mgr/dashboard/#dashboard-ssl-tls-support I see that the dashboards does support TLS certificates. But the Prometheus module apparently does not have this capability: https://docs.ceph.com/en/reef/mgr/prometheus/#prometheus-module
That's right. Docs need to be updated to reflect how to enable security and how the components are impacted.
So the prometheus mgr module actually does support TLS? Just to serve or also for client cert auth?
Is there any issue tracking these missing docs?
No, there's not. I'm going to open ticket this week (and probably also submit the docs PR).
Also the ceph-exporter distributed to Ceph node and exposes metrics for the local Ceph daemons. But it seems there is no authentication or TLS encryption support if you look at the code and the configuration options: https://github.com/ceph/ceph/blob/main/src/exporter/ceph_exporter.cc
That's a known issue. In case of small clusters you can disable ceph-exporter by using exclude_perf_counters configuration parameter: https://docs.ceph.com/en/latest/mgr/prometheus/#confval-mgr-prometheus-exclude_perf_counters
If this is a known issue, is there an issue or Trello card to track?
I had some discussions with the developer working on ceph-exporter about the lack of SSL/TLS and we agreed that he will be opening a tracker for that. However I can't find the issue. I'll double check with him.
Anyway, I'll recommend playing with the security feature by enabling the mgr/cephadm/secure_monitoring_stack and see if that fullfill your needs or not. It's relatively a new feature and any kind of feedback will be more than welcome.
Updated by Redouane Kachach Elhichou 2 months ago
- Related to Documentation #64596: secure monitoring stack support is not documented added
Updated by Christian Rohmann 25 days ago
Redouane Kachach Elhichou wrote in #note-5:
Christian Rohmann wrote:
Redouane Kachach Elhichou wrote:
So the prometheus mgr module actually does support TLS? Just to serve or also for client cert auth?
Is there any issue tracking these missing docs?No, there's not. I'm going to open ticket this week (and probably also submit the docs PR).
May I ask if you did you open a ticket about adding support for TLS (server and client certs) to mgr/prometheus?