Project

General

Profile

Feature #51947

cephadm: Redeploy services, on property update (was: Ingress for RGW does not appear to support chain certificates)

Added by Jim Bartlett over 1 year ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
cephadm
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

Using v16.2.4, Ubuntu 20.04 hosts for cluster and ingress (haproxy) for RGW instances. Multisite setup with one zone group, two zones. I have a certificate issued from an intermediate CA, which chains to a trusted root. In the certificate section of the ingress.yaml file, if I only put the server certificate, I will get errors trying to replicate as the secondary cannot build the certificate chain for TLS. I put the chain certificate in the ingress spec, but after applying and checking the haproxy.cfg, the chain has been stripped out and only the server certificate is left. Same TLS errors for replication. If I inject the chain certificate directly into the haproxy.cfg (bad idea of course, as it gets overwritten), TLS works properly as the secondary can build the chain back to the root. The ingress spec should support chain certificates as haproxy already does. The format below works in haproxy:

ssl_cert: |                         # optional: SSL certificate and key
    -----BEGIN PRIVATE KEY-----
    Key stuff
    -----END PRIVATE KEY-----
    -----BEGIN CERTIFICATE-----
    Server Certificate stuff
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
   Intermediate CA cert stuff
    -----END CERTIFICATE-----

Related issues

Related to Orchestrator - Feature #50061: cephadm: automatically redeploy daemons if user changes which container to use Closed
Duplicated by Orchestrator - Bug #54974: Applying rgwspec with new certificate does not apply New

History

#1 Updated by Dimitri Savineau over 1 year ago

That's weird because the code doesn't do anything special from the ssl_cert value in the spec

https://github.com/ceph/ceph/blob/v16.2.4/src/pybind/mgr/cephadm/services/ingress.py#L106-L110

So everything under ssl_cert should be written at the end in the haproxy.pem file the exact same way.

I've tested that small part of the code and the haproxy.pem file always has the right value.

I don't know if that's a typo when you pasted the ssl_cert value from the spec but keep in mind that you need two extra spaces before the data.

ssl_cert: | # optional: SSL certificate and key
  -----BEGIN PRIVATE KEY-----
  Key stuff
  -----END PRIVATE KEY-----
  -----BEGIN CERTIFICATE-----
  Server Certificate stuff
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  Intermediate CA cert stuff
  -----END CERTIFICATE-----

Can you share the full ingress spec file ?

#2 Updated by Dimitri Savineau over 1 year ago

I finished to test with v16.2.5 and I counldn't reproduce the issue.

---
service_type: ingress
service_id: object.ingress
placement:
  label: rgws
spec:
  backend_service: rgw.object
  virtual_ip: 192.168.100.100/24
  frontend_port: 8090
  monitor_port: 1967
  virtual_interface_networks:
    - 192.168.100.0/24
  ssl_cert: |
    -----BEGIN RSA PRIVATE KEY-----
    Key stuff
    -----END RSA PRIVATE KEY-----
    -----BEGIN CERTIFICATE-----
    Server Certificate stuff
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    Intermediate CA cert stuff
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    Root CA cert stuff
    -----END CERTIFICATE-----
...

And the haproxy.pem file generated has the right content.

cat /var/lib/ceph/7de08ebe-f0ad-11eb-9391-fa163eea1af0/haproxy.object.ingress.cephaio-1.yoeykn/haproxy/haproxy.pem
-----BEGIN RSA PRIVATE KEY-----
Key stuff
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
Server Certificate stuff
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
Intermediate CA cert stuff
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
Root CA cert stuff
-----END CERTIFICATE-----

#3 Updated by Dimitri Savineau over 1 year ago

Ok looks like you didn't redeploy the service after updating the spec file with the intermediate ca certificate right ?

I tried to deploy with certificate and key only.

Then I update the spec by adding the intermediate ca certificate and re-apply it.

Results:
- The spec is correctly updated in the KV store (ceph config-key get mgr/cephadm/spec.ingress.<service_id>)
- The haproxy certificate file on disk isn't updated (still certificate and key)

In fact I'm pretty sure that the normal behaviour because after updating the spec you need to redeploy the ingress services with

$ ceph orch redeploy ingress.<service_id>

After that the file on disk is updated.

Note that you could also remove the service and redeploy it from scratch.

#4 Updated by Jim Bartlett over 1 year ago

Ok, you're right, I did not redeploy, just re-applied the updated ingress yaml. I have tested on my newly upgraded 16.2.5 clusters and it is working as expected. Updating my procedures to include the re-deploy. I appreciate the help!

Jim.

#5 Updated by Sebastian Wagner over 1 year ago

  • Description updated (diff)

#6 Updated by Sebastian Wagner over 1 year ago

  • Tracker changed from Bug to Feature
  • Subject changed from Ingress for RGW does not appear to support chain certificates to cephadm: Redeploy services, on property update (was: Ingress for RGW does not appear to support chain certificates)
  • Category changed from cephadm/rgw to cephadm

#7 Updated by Sebastian Wagner about 1 year ago

  • Related to Feature #50061: cephadm: automatically redeploy daemons if user changes which container to use added

#8 Updated by Sebastian Wagner 10 months ago

  • Duplicated by Bug #54974: Applying rgwspec with new certificate does not apply added

Also available in: Atom PDF