Project

General

Profile

Bug #38614

Disable SSLv3 on ceph-mgr?

Added by A Boschke over 1 year ago. Updated 3 months ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
Category:
ceph-mgr
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

How does one configure ceph-mgr to disable SSLv3?

Ran NMAP's 'ssl-poodle' and it came up vulnerable to POODLE.
(nmap -sV --version-light --script ssl-poodle -p 8443 w.x.y.z)

We're using RPM ceph-mgr-13.2.4-0.el7.x86_64 from Ceph repository.

History

#1 Updated by Volker Theile over 1 year ago

What Python version are you using? In openSUSE Tumbleweed e.g. this is currently 2.7.15 and 3.7.2. I think most distributions use the latest 2.7 and 3.6 or 3.7 version. In this case i'm a little bit surprised because the default Python SSL implementation disables SSLv2 and SSLv3 by default:

https://github.com/python/cpython/blob/2.7/Lib/ssl.py#L427
https://github.com/python/cpython/blob/2.7/Modules/_ssl.c#L2245

https://github.com/python/cpython/blob/3.6/Lib/ssl.py#L499
https://github.com/python/cpython/blob/3.6/Modules/_ssl.c#L2841

https://github.com/python/cpython/blob/3.7/Lib/ssl.py#L562
https://github.com/python/cpython/blob/3.7/Modules/_ssl.c#L3032

#2 Updated by A Boschke over 1 year ago

I was surprised too.

Here are the specifics:
-

# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
# python --version
Python 2.7.5
# rpm -q python
python-2.7.5-76.el7.x86_64

#3 Updated by Volker Theile over 1 year ago

I've checked the content of python-2.7.5-76.el7.src.rpm and found out that the code Python code is from at least before Jan 9, 2014 (https://github.com/python/cpython/blob/3b2afbbf88cb8ba93542641f06c474aab13e50e6/Modules/_ssl.c). The feature to disable SSLv2 and SSLv3 was introduced by https://github.com/python/cpython/blob/daeb925cc88cc8fed2030166ade641de28edb396/Modules/_ssl.c. So finally, this package contains a Python version that is nearly 5 years old. IMHO the best is to upgrade to a more modern and secured Python version.

#4 Updated by Volker Theile over 1 year ago

Did some more research on the mentioned Python version in CentOS. The source package contains a patch (00214-pep466-backport-py3-ssl-changes.patch) which backports various py3 improvements, including disabled SSLv2 and SSLv3 protocols.

+def create_default_context(purpose=Purpose.SERVER_AUTH, cafile=None,
+                           capath=None, cadata=None):
+    """Create a SSLContext object with default settings.
+
+    NOTE: The protocol and settings may change anytime without prior
+          deprecation. The values represent a fair balance between maximum
+          compatibility and security.
+    """ 
+    if not isinstance(purpose, _ASN1Object):
+        raise TypeError(purpose)
+
+    context = SSLContext(PROTOCOL_SSLv23)
+
+    # SSLv2 considered harmful.
+    context.options |= OP_NO_SSLv2
+
+    # SSLv3 has problematic security and is only required for really old
+    # clients such as IE6 on Windows XP
+    context.options |= OP_NO_SSLv3

So i really do not have an idea why nmap find this vulnerability if these protocols are disabled.

#5 Updated by Ernesto Puerta over 1 year ago

A quick way to check this: openssl s_client -ssl3 -connect "$ip:$port"

#6 Updated by A Boschke over 1 year ago

So if I read the patch correctly, this sets the default SSL context for this object.
Is ceph-mgr changing the defaults or is nmap's test throwing a false positive?

(@Ernesto thanks for the troubleshooting tip)

#7 Updated by Volker Theile over 1 year ago

The dashboard does not use it's own context, so the SSL default context is used automatically.

#8 Updated by Ernesto Puerta over 1 year ago

  • Priority changed from Normal to Urgent
  • Severity changed from 3 - minor to 2 - major

Just saw a traceback in python "SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:618)" that led me to think that SSL3 is (somehow) not disabled. It might depend on my specific environment (as dev environment runs against pypi version and not distro's). I tested it with the above command and in fact my dashboard instance is responding to an SSLv3 context.

Overall, it seems that as we are depending on the Cherrypy builtin ssl module (module.py:194 config['server.ssl_module'] = 'builtin'), we don't benefit from OpenSSL security fixes.

A suggestion, partly following Volker's, would be to migrate to PyOpenSSL support:
- Proposed fix: https://stackoverflow.com/a/29304038
- Discussions: https://groups.google.com/forum/#!topic/cherrypy-users/ghrUwoyYKgI
- Related article: https://recollection.saaj.me/article/cherrypy-questions-testing-ssl-and-docker.html#ssl

Increased severity & priority.

#9 Updated by Kefu Chai about 1 year ago

  • Assignee set to Kefu Chai

#10 Updated by Kefu Chai about 1 year ago

it seems that as we are depending on the Cherrypy builtin ssl module (module.py:194 config['server.ssl_module'] = 'builtin'), we don't benefit from OpenSSL security fixes.

i doubt this. per https://docs.cherrypy.org/en/latest/deploy.html, "builtin" will use the builtin SSL in python, not the SSL module in cherrypy. also i checked https://github.com/cherrypy/cherrypy/blob/master/cherrypy/_cpwsgi_server.py#L85 , it is using https://github.com/cherrypy/cheroot/blob/8fd89e68dc31e62d3d583163f7d56b9ed7fde34c/cheroot/ssl/builtin.py for the "builtin" adaptor. where the context is created at https://github.com/cherrypy/cheroot/blob/8fd89e68dc31e62d3d583163f7d56b9ed7fde34c/cheroot/ssl/builtin.py#L97 . and the default options are used.

#11 Updated by Kefu Chai about 1 year ago

  • Assignee deleted (Kefu Chai)

#12 Updated by Kefu Chai about 1 year ago

  • Assignee set to Kefu Chai

#13 Updated by Kefu Chai about 1 year ago

$ MDS=0 MGR=1 OSD=3 MON=1 ../src/vstart.sh -n -x --memstore
...

dashboard urls: https://172.21.3.218:41626
  w/ user/pass: admin / admin
restful urls: https://172.21.3.218:42626
  w/ user/pass: admin / 8e949138-49a2-4cf6-ac7a-bc341ea31784

...

$ nmap -sV --version-light --script ssl-poodle -p 41626 172.21.3.218

Starting Nmap 7.60 ( https://nmap.org ) at 2019-10-21 10:07 UTC
Nmap scan report for incerta08.front.sepia.ceph.com (172.21.3.218)
Host is up (0.000080s latency).

PORT      STATE SERVICE VERSION
41626/tcp open  http    SABnzbd+ newsreader httpd

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 11.37 seconds
$ apt-cache show python-cherrypy3
Package: python-cherrypy3
Architecture: all
Version: 8.9.1-2
...
$ python --version
Python 2.7.15+
$ nmap -sV --version-light --script ssl-poodle -p 41262 192.168.2.8
Starting Nmap 7.80 ( https://nmap.org ) at 2019-10-21 19:41 CST
Nmap scan report for 192.168.2.8
Host is up (0.00011s latency).

PORT      STATE SERVICE  VERSION
41262/tcp open  ssl/http CherryPy wsgiserver

$ python3 --version
Python 3.7.5

$ apt-cache show python3-cherrypy3
Package: python3-cherrypy3
Source: cherrypy3
Version: 8.9.1-5
<pre>

will test on CentOS7.

#14 Updated by Sage Weil 10 months ago

  • Target version set to v15.0.0

#15 Updated by Lenz Grimmer 10 months ago

Is there anything for the dashboard to change here? I wonder if this is more of an environment/OS-specific issue?

#16 Updated by Sage Weil 10 months ago

I tried this against current master/octopus (containerized):

root@cpach:~# nmap -sV --version-light --script ssl-poodle -p 8443 10.3.64.20

Starting Nmap 7.60 ( https://nmap.org ) at 2020-01-31 16:53 CST
Stats: 0:00:13 elapsed; 0 hosts completed (1 up), 1 undergoing Service Scan
Service scan Timing: About 0.00% done
Nmap scan report for 10.3.64.20
Host is up (0.000090s latency).

PORT     STATE SERVICE  VERSION
8443/tcp open  ssl/http CherryPy httpd 18.4.0
MAC Address: 0C:C4:7A:D2:C7:70 (Super Micro Computer)

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 13.29 seconds

i think we're fine?

#17 Updated by Ernesto Puerta 10 months ago

Sage Weil wrote:

I tried this against current master/octopus (containerized):
[...]
i think we're fine?

Looks so (tested with python3-cherrypy-18.4.0-1.el8.noarch and python3-cheroot-8.2.1-1.el8.noarch) and no trace of SSLv3:

nmap --script ssl-enum-ciphers -p <port> <hostname>
Starting Nmap 7.70 ( https://nmap.org ) at 2020-02-03 10:26 CET
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000099s latency).
Other addresses for localhost (not scanned): ::1

PORT      STATE SERVICE
11013/tcp open  unknown
| ssl-enum-ciphers: 
|   TLSv1.2: 
|     ciphers: 
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (ecdh_x25519) - A
|       TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CCM (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CCM (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|     compressors: 
|       NULL
|     cipher preference: server
|_  least strength: A

And example of a positive (vulnerable) output can be found here

#18 Updated by Lenz Grimmer 10 months ago

  • Status changed from New to Can't reproduce
  • Target version deleted (v15.0.0)

This also does not seem to be an issue on openSUSE Leap 15.1 anymore. Resolving this as "Can't reproduce for now". Please re-open if you still observe this problem on any platform/version.

#19 Updated by A Boschke 5 months ago

Will a fix for this vulernabilty be back ported?

Currently, luminous, mimic, and nautilus are still listed as active.
In particular, we are still using mimic.

REF: [ https://docs.ceph.com/docs/master/releases/general/ ]

#20 Updated by Nathan Cutler 5 months ago

A Boschke wrote:

Will a fix for this vulernabilty be back ported?

Currently, luminous, mimic, and nautilus are still listed as active.
In particular, we are still using mimic.

REF: [ https://docs.ceph.com/docs/master/releases/general/ ]

It might be possible to backport it to mimic, but to know for sure I would need the SHA1 (or PR URL) of the fix.

#21 Updated by A Boschke 5 months ago

It seems the fix involves building with python3-cherrypy which for EL7 users seems unpackaged as of now.
(Naturally, we're using EL7 :)

#22 Updated by Nathan Cutler 5 months ago

A Boschke wrote:

It seems the fix involves building with python3-cherrypy which for EL7 users seems unpackaged as of now.
(Naturally, we're using EL7 :)

Are you sure we would need to build and release new Ceph packages for that, though? Wouldn't it be enough to update your "python2-cherrypy" or "python-cherrypy" package (the one you have installed on the Ceph cluster nodes) to a version containing the fix?

NOTE: Mimic uses Python 2, so "python3-cherrypy" would not help there.

#23 Updated by A Boschke 5 months ago

Your question is best answered by developers who did that actual troubleshooting.

#24 Updated by Ernesto Puerta 5 months ago

A Boschke wrote:

Your question is best answered by developers who did that actual troubleshooting.

ceph-mgr packages are not pinned to any specific CherryPy version, so the only thing you need to do is to install a fixed Cherrpy package from your distro provider. If not available, Cherrypy/Cherroot is a pure-Python package, so you could simply install it from Pypi.

#25 Updated by A Boschke 5 months ago

My followup question here really should have been worded differently.

Will the supported Mimic repository's ceph-mgr package be rebuilt so that SSLv3 is disabled?
It's still enabled, yet supported by the project.

If it won't be fixed, we'll the package at least me amended to include a warning about POODLE et al?

#26 Updated by Ernesto Puerta 5 months ago

A Boschke wrote:

My followup question here really should have been worded differently.

Will the supported Mimic repository's ceph-mgr package be rebuilt so that SSLv3 is disabled?
It's still enabled, yet supported by the project.

If it won't be fixed, we'll the package at least me amended to include a warning about POODLE et al?

I understood what you meant, but when there's a vulnerability in a package the usual practice is to fix the vulnerability in the compromised package and release that very one, not rebuilding or adding warnings to everything else just to work around the compromised piece of code.

In Mimic we were already using Python ssl module, so that moves the focus to Python. In ssl module, since Python 2.7, the default behavior is to reject SSLv3, so unless I'm missing anything here: this would be fixed (in fact no code change has been delivered here).

Are you still experiencing this issue?

#27 Updated by Willem Jan Withagen 3 months ago

Ernesto Puerta wrote:

In Mimic we were already using Python ssl module, so that moves the focus to Python. In ssl module, since Python 2.7, the default behavior is to reject SSLv3, so unless I'm missing anything here: this would be fixed (in fact no code change has been delivered here).

Are you still experiencing this issue?

I'm actually having this on FreeBSD with Nautilus.
But this track leads be to the assumption that Ceph is not at fault here.
Somewhere in the Python stack SSLv3 is not disabled, and that causes the problem.
Which boils down to a FreeBSD packaging configuration thingy.

I'll try the open_ssl test suggested, and track from there

#28 Updated by Willem Jan Withagen 3 months ago

Willem Jan Withagen wrote:

Ernesto Puerta wrote:

In Mimic we were already using Python ssl module, so that moves the focus to Python. In ssl module, since Python 2.7, the default behavior is to reject SSLv3, so unless I'm missing anything here: this would be fixed (in fact no code change has been delivered here).

Are you still experiencing this issue?

I'm actually having this on FreeBSD with Nautilus.
But this track leads be to the assumption that Ceph is not at fault here.
Somewhere in the Python stack SSLv3 is not disabled, and that causes the problem.
Which boils down to a FreeBSD packaging configuration thingy.

I'll try the open_ssl test suggested, and track from there

This is what I get from a connect with openssl ssl3:
CONNECTED
34371092480:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:/usr/src/crypto/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 40
--

no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 58 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : SSLv3
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1599390869
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
Exit 1 ========================================

So that does not seem to work.

Giving openssl the free hand, does work and make a TLS 1.3 link.
Now this is on a server where openssl no longer has ssl2/3 compiled in.
So that sort of fixes the problem in a practical way.

But the alternative would be to actually set up the ciphers in the webserver.
Newer CherryPy and likes have provisions for that.

  1. openssl s_client connect 192.168.11.201:8081
    CONNECTED
    Can't use SSL_get_servername
    depth=0 O = IT, CN = ceph-dashboard
    verify error:num=18:self signed certificate
    verify return:1
    depth=0 O = IT, CN = ceph-dashboard
    verify return:1
    --

    Certificate chain
    0 s:O = IT, CN = ceph-dashboard
    i:O = IT, CN = ceph-dashboard
    ---
    Server certificate
    -----BEGIN CERTIFICATE-----
    ..............
    -----END CERTIFICATE-----
    subject=O = IT, CN = ceph-dashboard

issuer=O = IT, CN = ceph-dashboard

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1279 bytes and written 375 bytes
Verification error: self signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self signed certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 149A67F50BEBC0F4E089DDEC191C132FC76C0105F05967058F6621EBF4E519A4
Session-ID-ctx:
Resumption PSK: 7C2CCFF4A52D4191B50C542F0C80981A4E6A7C900C61745D0CE3D903B57C0CF6FF7098AAB5B7181C3A4F22863AF3670E
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
...............
Start Time: 1599390927
Timeout : 7200 (sec)
Verify return code: 18 (self signed certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 018BB782E102B854CF3A555C9A08CF3883D9C97753C6B4C97445F034A4428D0F
Session-ID-ctx:
Resumption PSK: BA49EFB8D7A7B2833052841F3E15B3EF1874F6E60FBB800CA2560C3300F1785F14D841E4A4C881676A2FE5AA6BF6FFAA
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
..............
Start Time: 1599390927
Timeout : 7200 (sec)
Verify return code: 18 (self signed certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
HTTP/1.1 408 Request Timeout
Content-Length: 0
Content-Type: text/plain

read:errno=0

#29 Updated by Ernesto Puerta 3 months ago

Hi Willem, as mentioned we couldn't reproduce this issue in the distros we use so there's no way to validate some of the workarounds discussed, but as you managed to reproduce it, feel free to send a PR with the patch and we'll review that.

Also available in: Atom PDF