Project

General

Profile

Bug #57924

mgr/dashboard: fails with "Module 'dashboard' has failed: key type unsupported" when using letsencrypt ec certificates

Added by Björn Lässig 3 months ago. Updated 3 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Description of problem

After generating a recent certificate by letsencrypt and configuring dashboard to use them, dashboard is not working anymore.

Environment

  • ceph version string: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
  • Platform (OS/distro/release): Debian
  • Cluster details (nodes, monitors, OSDs): 30 OSDs on 4 Nodes, 5 Monitors, 2 Managers
  • Did it happen on a stable environment or after a migration/upgrade?: happened on stable
  • Browser used (e.g.: Version 86.0.4240.198 (Official Build) (64-bit)): irrelevant. There was no service listening anymore.

How reproducible

Steps:

  1. generate key with dehydrated
  2. use keys for dashboard
    [root@ceph:~] 22 # ceph dashboard set-ssl-certificate-key -i /var/lib/dehydrated/certs/local/privkey.pem
    SSL certificate key updated
    [root@ceph:~] # ceph dashboard set-ssl-certificate -i /var/lib/dehydrated/certs/local/fullchain.pem
    SSL certificate updated
    
  1. Restart Manager

    ceph mgr fail

-> Now there is no dashboard anymore.

Actual results

Okt 25 14:13:06 cephmgr2 ceph-mgr[1039008]: [prometheus INFO cherrypy.error] [25/Oct/2022:12:13:06] ENGINE Bus STARTING
Okt 25 14:13:06 cephmgr2 ceph-mgr[1039008]: log_channel(cluster) log [ERR] : Unhandled exception from module 'dashboard' while running on mgr.cephmgr2.zvtgjh: key type unsupported
Okt 25 14:13:06 cephmgr2 ceph-mgr[1039008]: dashboard.serve:
Okt 25 14:13:06 cephmgr2 conmon[1039003]: 2022-10-25T12:13:06.163+0000 7fc726376700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'dashboard' while running on mgr.cephmgr2.zvtgjh: key type unsupported
Okt 25 14:13:06 cephmgr2 conmon[1039003]: 2022-10-25T12:13:06.167+0000 7fc726376700 -1 dashboard.serve:
Okt 25 14:13:06 cephmgr2 conmon[1039003]: 2022-10-25T12:13:06.167+0000 7fc726376700 -1 Traceback (most recent call last):
Okt 25 14:13:06 cephmgr2 conmon[1039003]:   File "/usr/share/ceph/mgr/dashboard/module.py", line 508, in serve
Okt 25 14:13:06 cephmgr2 conmon[1039003]:     uri = self.await_configuration()
Okt 25 14:13:06 cephmgr2 conmon[1039003]:   File "/usr/share/ceph/mgr/dashboard/module.py", line 211, in await_configuration
Okt 25 14:13:06 cephmgr2 conmon[1039003]:     uri = self._configure()
Okt 25 14:13:06 cephmgr2 conmon[1039003]:   File "/usr/share/ceph/mgr/dashboard/module.py", line 172, in _configure
Okt 25 14:13:06 cephmgr2 conmon[1039003]:     verify_tls_files(cert_fname, pkey_fname)
Okt 25 14:13:06 cephmgr2 conmon[1039003]:   File "/usr/share/ceph/mgr/mgr_util.py", line 638, in verify_tls_files
Okt 25 14:13:06 cephmgr2 conmon[1039003]:     pkey.check()
Okt 25 14:13:06 cephmgr2 conmon[1039003]:   File "/lib/python3.6/site-packages/OpenSSL/crypto.py", line 344, in check
Okt 25 14:13:06 cephmgr2 conmon[1039003]:     raise TypeError("key type unsupported")
Okt 25 14:13:06 cephmgr2 conmon[1039003]: TypeError: key type unsupported
Okt 25 14:13:06 cephmgr2 conmon[1039003]: 
Okt 25 14:13:06 cephmgr2 ceph-mgr[1039008]: Traceback (most recent call last):
                                              File "/usr/share/ceph/mgr/dashboard/module.py", line 508, in serve
                                                uri = self.await_configuration()
                                              File "/usr/share/ceph/mgr/dashboard/module.py", line 211, in await_configuration
                                                uri = self._configure()
                                              File "/usr/share/ceph/mgr/dashboard/module.py", line 172, in _configure
                                                verify_tls_files(cert_fname, pkey_fname)
                                              File "/usr/share/ceph/mgr/mgr_util.py", line 638, in verify_tls_files
                                                pkey.check()
                                              File "/lib/python3.6/site-packages/OpenSSL/crypto.py", line 344, in check
                                                raise TypeError("key type unsupported")
                                            TypeError: key type unsupported

# ceph status

  cluster:
    id:     XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
    health: HEALTH_ERR
            Module 'dashboard' has failed: key type unsupported

there was no service on port 8443 listening.

Expected results

A Dashboard with a valid Certificate

Additional info

Workaround:

  1. remove cert/key when manager dashboard does not work:
# ceph config-key rm mgr/dashboard/cert                                                                             
key deleted
# ceph config-key rm mgr/dashboard/key
key deleted

# force dehydrated to use rsa private key:
dehydrated --algo rsa  -x -c 

# now use the rsa cert
cd /var/lib/dehydrated/certs/local/
ceph config-key set mgr/dashboard/crt -i fullchain.pem
ceph config-key set mgr/dashboard/key -i privkey.pem.pem

History

#1 Updated by Björn Lässig 3 months ago

Certificate that did not work:

# openssl x509 -text -in  fullchain-1666692624.pem
Certificate:                                                         
    Data:                                                                                                                                 
        Version: 3 (0x2)                                             
        Serial Number:                                                                                                                    
            XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity                                                      
            Not Before: Oct 25 09:10:37 2022 GMT                      
            Not After : Jan 23 09:10:36 2023 GMT                      
        Subject: CN = <ceph.fqdn>                         
        Subject Public Key Info:                                      
            Public Key Algorithm: id-ecPublicKey                      
                Public-Key: (384 bit)                                 
                pub:                                                  
                    04:18:71:6c:af:43:3b:e2:29:4a:96:f7:f3:94:c4:     
                    17:1b:cf:46:50:48:b5:11:7c:79:7c:a1:8d:68:80:     
                    93:4c:3c:90:31:75:8a:3b:8a:cf:66:48:08:bd:46:     
                    2a:18:e6:6d:6d:d7:f9:81:bd:c6:a6:9d:e9:7b:2b:     
                    38:57:f8:fb:ee:80:24:9f:ae:21:31:ac:a3:ba:6d:     
                    94:c3:5d:52:c5:f8:1c:6d:af:ca:9b:3b:0c:b0:21:     
                    7a:19:b3:e3:c9:7e:72                              
                ASN1 OID: secp384r1                                   
                NIST CURVE: P-384                                     

Certificate:                                                                                                                                                                                                                                                                              
    Data:                                                                                                                                                                                                                                                                                 
        Version: 3 (0x2)                                                                                                                                                                                                                                                                  
        Serial Number:                                                                                                                                                                                                                                                                    
            YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY:YY
        Signature Algorithm: sha256WithRSAEncryption                                                                                                                                                                                                                                      
        Issuer: C = US, O = Let's Encrypt, CN = R3                                                                                                                                                                                                                                        
        Validity                                                                                                                                                                                                                                                                          
            Not Before: Oct 25 11:45:26 2022 GMT                                                                                                                                                                                                                                          
            Not After : Jan 23 11:45:25 2023 GMT                                                                                                                                                                                                                                          
        Subject: CN = !ceph.fqdn>                                                                                                                                                                                                                                             
        Subject Public Key Info:                                                                                                                                                                                                                                                          
            Public Key Algorithm: rsaEncryption                                                                                                                                                                                                                                           
                RSA Public-Key: (4096 bit)                                                                                                                                                                                                                                                
                Modulus:                                                                                                                                                                                                                                                                  
                    00:lotsofbyptes:XX:
                    XX:XX:XX
                Exponent: 65537 (0x10001)

#2 Updated by Björn Lässig 3 months ago

the 2nd quoted cert was working. Unfortunately i cannot fix my own bug reports.

Also available in: Atom PDF