Actions
Bug #59703
openCertificate renewal process failed
Status:
New
Priority:
Normal
Assignee:
-
Category:
msgr
Target version:
-
% Done:
0%
Source:
other
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Description
Hi Hi,
We upgraded our cluster from debian 11 to debian 12, then upgraded ceph from `v14.2.21-1` to `v16.2.11` using the debian repos.
Then when renewing the Let's encrypt SSL cert used by the dashboard and importing it again we ran into the following issue:
18:10:22 pbackup01.XX ceph-mgr[2447]: 2023-05-09T18:23:34.876+0200 7f4db64b0440 -1 mgr[py] Traceback (most recent call last):
18:10:22 pbackup01.XX ceph-mgr[2447]: File "/usr/share/ceph/mgr/restful/__init__.py", line 1, in <module>
18:10:22 pbackup01.XX ceph-mgr[2447]: from .module import Module
18:10:22 pbackup01.XX ceph-mgr[2447]: File "/usr/share/ceph/mgr/restful/module.py", line 22, in <module>
18:10:22 pbackup01.XX ceph-mgr[2447]: from OpenSSL import crypto
18:10:22 pbackup01.XX ceph-mgr[2447]: File "/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
18:10:22 pbackup01.XX ceph-mgr[2447]: from OpenSSL import SSL, crypto
18:10:22 pbackup01.XX ceph-mgr[2447]: File "/lib/python3/dist-packages/OpenSSL/SSL.py", line 19, in <module>
18:10:22 pbackup01.XX ceph-mgr[2447]: from OpenSSL.crypto import (
18:10:22 pbackup01.XX ceph-mgr[2447]: File "/lib/python3/dist-packages/OpenSSL/crypto.py", line 21, in <module>
18:10:22 pbackup01.XX ceph-mgr[2447]: from cryptography import utils, x509
18:10:22 pbackup01.XX ceph-mgr[2447]: File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 6, in <module>
18:10:22 pbackup01.XX ceph-mgr[2447]: from cryptography.x509 import certificate_transparency
18:10:22 pbackup01.XX ceph-mgr[2447]: File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py", line 10, in <module>
18:10:22 pbackup01.XX ceph-mgr[2447]: from cryptography.hazmat.bindings._rust import x509 as rust_x509
18:10:22 pbackup01.XX ceph-mgr[2447]: ImportError: PyO3 modules may only be initialized once per interpreter process
...
18:10:27 pbackup01.XX ceph-mgr[2447]: File "/usr/share/ceph/mgr/restful/__init__.py", line 1, in <module>
18:10:27 pbackup01.XX ceph-mgr[2447]: from .module import Module
18:10:27 pbackup01.XX ceph-mgr[2447]: File "/usr/share/ceph/mgr/restful/module.py", line 22, in <module>
18:10:27 pbackup01.XX ceph-mgr[2447]: from OpenSSL import crypto
18:10:27 pbackup01.XX ceph-mgr[2447]: File "/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
18:10:27 pbackup01.XX ceph-mgr[2447]: from OpenSSL import SSL, crypto
18:10:27 pbackup01.XX ceph-mgr[2447]: File "/lib/python3/dist-packages/OpenSSL/SSL.py", line 19, in <module>
18:10:27 pbackup01.XX ceph-mgr[2447]: from OpenSSL.crypto import (
18:10:27 pbackup01.XX ceph-mgr[2447]: File "/lib/python3/dist-packages/OpenSSL/crypto.py", line 21, in <module>
18:10:27 pbackup01.XX ceph-mgr[2447]: from cryptography import utils, x509
18:10:27 pbackup01.XX ceph-mgr[2447]: File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 6, in <module>
18:10:27 pbackup01.XX ceph-mgr[2447]: from cryptography.x509 import certificate_transparency
18:10:27 pbackup01.XX ceph-mgr[2447]: File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py", line 10, in <module>
18:10:27 pbackup01.XX ceph-mgr[2447]: from cryptography.hazmat.bindings._rust import x509 as rust_x509
18:10:27 pbackup01.XX ceph-mgr[2447]: ImportError: PyO3 modules may only be initialized once per interpreter process
18:10:27 pbackup01.XX ceph-mgr[2449]: -1 log_channel(cluster) log [ERR] : Health check failed: Module 'dashboard' has failed: Only RSA keys can currently be checked. (MGR_MODULE_ERROR)
I've double-checked the certificate and it's indeed a valid one:
sudo openssl x509 -in /etc/letsencrypt/live/pbackup01.XX/fullchain.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
03:3b:44:9a:64:d0:3f:f4:2a:e3:46:dc:00:4e:e6:27:a7:fd
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = R3
Validity
Not Before: May 9 07:50:52 2023 GMT
Not After : Aug 7 07:50:51 2023 GMT
Subject: CN = pbackup01.XX
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:f4:2d:1f:d8:db:65:6f:37:b2:3d:6f:00:c2:d0:
22:01:84:f3:a6:11:de:10:54:76:fd:05:99:9a:cc:
d4:c1:12:45:92:f1:39:f7:9f:e7:4f:a6:60:fe:9b:
1f:1d:30:8f:26:e4:8c:09:3e:96:a0:76:6d:6d:1b:
17:ce:57:04:f3
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
84:6C:CD:DD:D8:F5:7E:F0:EA:79:B2:95:FD:32:6C:56:41:02:F7:A5
X509v3 Authority Key Identifier:
14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6
Authority Information Access:
OCSP - URI:http://r3.o.lencr.org
CA Issuers - URI:http://r3.i.lencr.org/
X509v3 Subject Alternative Name:
DNS:pbackup01bsl01.sys.init7.net
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.letsencrypt.org
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : B7:3E:FB:24:DF:9C:4D:BA:75:F2:39:C5:BA:58:F4:6C:
5D:FC:42:CF:7A:9F:35:C4:9E:1D:09:81:25:ED:B4:99
Timestamp : May 9 08:50:52.868 2023 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:D9:4F:86:7F:48:38:54:76:BA:0F:B0:
65:76:CC:9F:C5:94:C9:AE:9F:A2:25:72:B4:42:94:D1:
AA:43:87:F2:24:02:21:00:B1:63:2B:9E:0C:21:86:73:
3E:66:9E:5B:2B:2D:94:6E:0F:85:AA:1B:C5:00:E0:C7:
DE:F6:24:CE:BE:7E:01:B3
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 7A:32:8C:54:D8:B7:2D:B6:20:EA:38:E0:52:1E:E9:84:
16:70:32:13:85:4D:3B:D2:2B:C1:3A:57:A3:52:EB:52
Timestamp : May 9 08:50:53.387 2023 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:F1:86:53:10:AA:8A:99:D5:BB:CC:97:
F8:1D:D7:49:77:1E:F0:D7:63:35:F1:27:F6:00:DB:10:
1C:94:16:FB:E6:02:21:00:86:CD:61:FB:F0:BC:5F:DE:
F9:AF:35:42:A8:A4:36:DF:81:42:2A:93:79:6B:66:EC:
1E:47:C2:FA:ED:27:7B:A5
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
01:f5:d8:2f:9d:b0:e5:eb:72:39:cd:23:f7:2b:ab:51:7a:39:
32:e0:f4:20:d8:10:7b:40:69:85:1e:bd:94:58:27:78:c1:fd:
74:d0:9a:3a:9f:7a:89:5b:dd:12:76:53:f1:ac:f3:ad:60:3f:
4c:68:94:be:9c:c1:23:a1:e7:58:a1:8b:32:21:9b:01:32:31:
ab:d8:b4:d8:d2:e4:c5:72:31:1f:fb:2c:a6:b6:60:01:ac:79:
9c:cb:9c:11:bf:96:02:cf:89:71:6d:d4:48:ce:65:5f:8a:e4:
a7:44:b9:5a:52:28:7f:9c:f8:c5:ed:e0:7b:8c:a2:d6:d8:9e:
ff:4f:2c:2a:94:2e:1b:9c:cc:c2:4e:1d:6a:0c:b7:5b:a8:51:
73:fe:de:d9:16:bf:6a:a3:26:ba:ac:68:f2:2e:51:ab:0c:34:
8d:5e:2d:17:b3:cb:fd:c9:c6:02:fb:18:78:68:89:09:19:9c:
90:1e:66:5e:eb:8d:49:b8:6a:86:78:c1:3f:97:c0:1a:39:30:
9d:fb:d5:c3:ea:d6:be:25:75:3d:6d:61:62:8a:eb:aa:30:41:
30:dd:d1:23:1e:ca:ae:53:27:d9:95:31:81:5e:8d:e0:07:7a:
a7:7f:54:aa:19:33:e6:49:38:c0:e3:cb:6d:d4:bc:24:35:3f:
d7:ce:de:1f
The renewal process has been done as described:
sudo ceph dashboard set-ssl-certificate-key -i /etc/letsencrypt/live/pbackup01.XX/privkey.pem
sudo ceph dashboard set-ssl-certificate -i /etc/letsencrypt/live/pbackup01.XX/fullchain.pem
sudo ceph mgr module disable dashboard
sudo ceph mgr module enable dashboard
After this point the only way to have the dashboard module working again was to proceed as following:
sudo ceph config-key set mgr/dashboard/crt #empty to flush the content
sudo ceph config-key set mgr/dashboard/key #empty to flush the content
sudo ceph mgr module disable dashboard
sudo ceph mgr module enable dashboard
sudo ceph dashboard create-self-signed-cert
sudo ceph mgr module disable dashboard
sudo ceph mgr module enable dashboard
sudo ceph mgr services
{
"dashboard": "https://XX.XX.XX.XX:8443/"
}
Updated by Peter Pavlisko 10 months ago
I would like to point out that the error message is confusing and completely not helpful:
ImportError: PyO3 modules may only be initialized once per interpreter process
This is what I was greeted after ceph upgrade. It took me two days to arrive here and realize that this is probably the same SSL related issue.
Actions