Project

General

Profile

Actions

Feature #64318

open

mgr/prometheus add support for TLS and client cert authentication

Added by Christian Rohmann 3 months ago. Updated 15 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
prometheus module
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

Support for using TLS in all sorts of ways has come to a lot of (written in Golang) exporters via the exporter-toolkit (https://github.com/prometheus/exporter-toolkit/blob/master/CHANGELOG.md) or via other ways.

Looking at https://docs.ceph.com/en/latest/mgr/prometheus/#configuration, the Prometheus module of the mgr does not have any support for using TLS.
Be it for the HTTP server itself or to authenticate clients with client certificates. Both of which Prometheus can quite easily make use of by setting tls_config (https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config) within the scrape config.

It would be good to be able to protect Ceph metrics endpoints via TLS transport encryption and also client certificates.
The same actually also applies to the ceph-exporter (https://github.com/ceph/ceph/tree/main/src/exporter).

There is an (unofficial) client library for C++ at https://github.com/jupp0r/prometheus-cpp, I don't know this makes sense to use for Ceph.


Related issues 1 (0 open1 closed)

Related to Orchestrator - Documentation #64596: secure monitoring stack support is not documentedResolvedRedouane Kachach Elhichou

Actions
Actions #1

Updated by Redouane Kachach Elhichou 2 months ago

cephadm has already support to enable security across all the monitoring stack (including all the components). The configuration variable is mgr/cephadm/secure_monitoring_stack maybe it's just a documentation effort in this case.

Actions #2

Updated by Christian Rohmann 2 months ago

Redouane Kachach Elhichou wrote:

cephadm has already support to enable security across all the monitoring stack (including all the components). The configuration variable is mgr/cephadm/secure_monitoring_stack maybe it's just a documentation effort in this case.

I am not sure really. What does this setting actually do for the various components?

Looking at https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/services/monitoring.py it appears it's using https to reach the service_discovery of the mgr or adds authentication to Prometheus + Alertmanager.

But is there any TLS added? Looking at https://docs.ceph.com/en/reef/mgr/dashboard/#dashboard-ssl-tls-support I see that the dashboards does support TLS certificates. But the Prometheus module apparently does not have this capability: https://docs.ceph.com/en/reef/mgr/prometheus/#prometheus-module

Also the ceph-exporter distributed to Ceph node and exposes metrics for the local Ceph daemons. But it seems there is no authentication or TLS encryption support if you look at the code and the configuration options: https://github.com/ceph/ceph/blob/main/src/exporter/ceph_exporter.cc

Actions #3

Updated by Redouane Kachach Elhichou about 2 months ago

Christian Rohmann wrote:

Redouane Kachach Elhichou wrote:

cephadm has already support to enable security across all the monitoring stack (including all the components). The configuration variable is mgr/cephadm/secure_monitoring_stack maybe it's just a documentation effort in this case.

I am not sure really. What does this setting actually do for the various components?

It enables security (SSL/TLS and basic auth) for all the monitoring components.

Looking at https://github.com/ceph/ceph/blob/main/src/pybind/mgr/cephadm/services/monitoring.py it appears it's using https to reach the service_discovery of the mgr or adds authentication to Prometheus + Alertmanager.

But is there any TLS added? Looking at https://docs.ceph.com/en/reef/mgr/dashboard/#dashboard-ssl-tls-support I see that the dashboards does support TLS certificates. But the Prometheus module apparently does not have this capability: https://docs.ceph.com/en/reef/mgr/prometheus/#prometheus-module

That's right. Docs need to be updated to reflect how to enable security and how the components are impacted.

Also the ceph-exporter distributed to Ceph node and exposes metrics for the local Ceph daemons. But it seems there is no authentication or TLS encryption support if you look at the code and the configuration options: https://github.com/ceph/ceph/blob/main/src/exporter/ceph_exporter.cc

That's a known issue. In case of small clusters you can disable ceph-exporter by using exclude_perf_counters configuration parameter: https://docs.ceph.com/en/latest/mgr/prometheus/#confval-mgr-prometheus-exclude_perf_counters

Actions #4

Updated by Christian Rohmann about 2 months ago

Redouane Kachach Elhichou wrote:

But is there any TLS added? Looking at https://docs.ceph.com/en/reef/mgr/dashboard/#dashboard-ssl-tls-support I see that the dashboards does support TLS certificates. But the Prometheus module apparently does not have this capability: https://docs.ceph.com/en/reef/mgr/prometheus/#prometheus-module

That's right. Docs need to be updated to reflect how to enable security and how the components are impacted.

So the prometheus mgr module actually does support TLS? Just to serve or also for client cert auth?
Is there any issue tracking these missing docs?

Also the ceph-exporter distributed to Ceph node and exposes metrics for the local Ceph daemons. But it seems there is no authentication or TLS encryption support if you look at the code and the configuration options: https://github.com/ceph/ceph/blob/main/src/exporter/ceph_exporter.cc

That's a known issue. In case of small clusters you can disable ceph-exporter by using exclude_perf_counters configuration parameter: https://docs.ceph.com/en/latest/mgr/prometheus/#confval-mgr-prometheus-exclude_perf_counters

If this is a known issue, is there an issue or Trello card to track?

Actions #5

Updated by Redouane Kachach Elhichou about 2 months ago

Christian Rohmann wrote:

Redouane Kachach Elhichou wrote:

But is there any TLS added? Looking at https://docs.ceph.com/en/reef/mgr/dashboard/#dashboard-ssl-tls-support I see that the dashboards does support TLS certificates. But the Prometheus module apparently does not have this capability: https://docs.ceph.com/en/reef/mgr/prometheus/#prometheus-module

That's right. Docs need to be updated to reflect how to enable security and how the components are impacted.

So the prometheus mgr module actually does support TLS? Just to serve or also for client cert auth?
Is there any issue tracking these missing docs?

No, there's not. I'm going to open ticket this week (and probably also submit the docs PR).

Also the ceph-exporter distributed to Ceph node and exposes metrics for the local Ceph daemons. But it seems there is no authentication or TLS encryption support if you look at the code and the configuration options: https://github.com/ceph/ceph/blob/main/src/exporter/ceph_exporter.cc

That's a known issue. In case of small clusters you can disable ceph-exporter by using exclude_perf_counters configuration parameter: https://docs.ceph.com/en/latest/mgr/prometheus/#confval-mgr-prometheus-exclude_perf_counters

If this is a known issue, is there an issue or Trello card to track?

I had some discussions with the developer working on ceph-exporter about the lack of SSL/TLS and we agreed that he will be opening a tracker for that. However I can't find the issue. I'll double check with him.

Anyway, I'll recommend playing with the security feature by enabling the mgr/cephadm/secure_monitoring_stack and see if that fullfill your needs or not. It's relatively a new feature and any kind of feedback will be more than welcome.

Actions #6

Updated by Redouane Kachach Elhichou about 2 months ago

Actions #7

Updated by Christian Rohmann 15 days ago

Redouane Kachach Elhichou wrote in #note-5:

Christian Rohmann wrote:

Redouane Kachach Elhichou wrote:

So the prometheus mgr module actually does support TLS? Just to serve or also for client cert auth?
Is there any issue tracking these missing docs?

No, there's not. I'm going to open ticket this week (and probably also submit the docs PR).

May I ask if you did you open a ticket about adding support for TLS (server and client certs) to mgr/prometheus?

Actions

Also available in: Atom PDF