Ceph » mgr

What Python version are you using? In openSUSE Tumbleweed e.g. this is currently 2.7.15 and 3.7.2. I think most distributions use the latest 2.7 and 3.6 or 3.7 version. In this case i'm a little bit surprised because the default Python SSL implementation disables SSLv2 and SSLv3 by default:

https://github.com/python/cpython/blob/2.7/Lib/ssl.py#L427
https://github.com/python/cpython/blob/2.7/Modules/_ssl.c#L2245

https://github.com/python/cpython/blob/3.6/Lib/ssl.py#L499
https://github.com/python/cpython/blob/3.6/Modules/_ssl.c#L2841

https://github.com/python/cpython/blob/3.7/Lib/ssl.py#L562
https://github.com/python/cpython/blob/3.7/Modules/_ssl.c#L3032

I was surprised too.

Here are the specifics:
-

# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
# python --version
Python 2.7.5
# rpm -q python
python-2.7.5-76.el7.x86_64

I've checked the content of python-2.7.5-76.el7.src.rpm and found out that the code Python code is from at least before Jan 9, 2014 (https://github.com/python/cpython/blob/3b2afbbf88cb8ba93542641f06c474aab13e50e6/Modules/_ssl.c). The feature to disable SSLv2 and SSLv3 was introduced by https://github.com/python/cpython/blob/daeb925cc88cc8fed2030166ade641de28edb396/Modules/_ssl.c. So finally, this package contains a Python version that is nearly 5 years old. IMHO the best is to upgrade to a more modern and secured Python version.

Did some more research on the mentioned Python version in CentOS. The source package contains a patch (00214-pep466-backport-py3-ssl-changes.patch) which backports various py3 improvements, including disabled SSLv2 and SSLv3 protocols.

+def create_default_context(purpose=Purpose.SERVER_AUTH, cafile=None,
+                           capath=None, cadata=None):
+    """Create a SSLContext object with default settings.
+
+    NOTE: The protocol and settings may change anytime without prior
+          deprecation. The values represent a fair balance between maximum
+          compatibility and security.
+    """ 
+    if not isinstance(purpose, _ASN1Object):
+        raise TypeError(purpose)
+
+    context = SSLContext(PROTOCOL_SSLv23)
+
+    # SSLv2 considered harmful.
+    context.options |= OP_NO_SSLv2
+
+    # SSLv3 has problematic security and is only required for really old
+    # clients such as IE6 on Windows XP
+    context.options |= OP_NO_SSLv3

So i really do not have an idea why nmap find this vulnerability if these protocols are disabled.

A quick way to check this: openssl s_client -ssl3 -connect "$ip:$port"

So if I read the patch correctly, this sets the default SSL context for this object.
Is ceph-mgr changing the defaults or is nmap's test throwing a false positive?

(@Ernesto thanks for the troubleshooting tip)

The dashboard does not use it's own context, so the SSL default context is used automatically.

it seems that as we are depending on the Cherrypy builtin ssl module (module.py:194 config['server.ssl_module'] = 'builtin'), we don't benefit from OpenSSL security fixes.

i doubt this. per https://docs.cherrypy.org/en/latest/deploy.html, "builtin" will use the builtin SSL in python, not the SSL module in cherrypy. also i checked https://github.com/cherrypy/cherrypy/blob/master/cherrypy/_cpwsgi_server.py#L85 , it is using https://github.com/cherrypy/cheroot/blob/8fd89e68dc31e62d3d583163f7d56b9ed7fde34c/cheroot/ssl/builtin.py for the "builtin" adaptor. where the context is created at https://github.com/cherrypy/cheroot/blob/8fd89e68dc31e62d3d583163f7d56b9ed7fde34c/cheroot/ssl/builtin.py#L97 . and the default options are used.

$ MDS=0 MGR=1 OSD=3 MON=1 ../src/vstart.sh -n -x --memstore
...

dashboard urls: https://172.21.3.218:41626
  w/ user/pass: admin / admin
restful urls: https://172.21.3.218:42626
  w/ user/pass: admin / 8e949138-49a2-4cf6-ac7a-bc341ea31784

...

$ nmap -sV --version-light --script ssl-poodle -p 41626 172.21.3.218

Starting Nmap 7.60 ( https://nmap.org ) at 2019-10-21 10:07 UTC
Nmap scan report for incerta08.front.sepia.ceph.com (172.21.3.218)
Host is up (0.000080s latency).

PORT      STATE SERVICE VERSION
41626/tcp open  http    SABnzbd+ newsreader httpd

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 11.37 seconds

$ apt-cache show python-cherrypy3
Package: python-cherrypy3
Architecture: all
Version: 8.9.1-2
...
$ python --version
Python 2.7.15+

$ nmap -sV --version-light --script ssl-poodle -p 41262 192.168.2.8
Starting Nmap 7.80 ( https://nmap.org ) at 2019-10-21 19:41 CST
Nmap scan report for 192.168.2.8
Host is up (0.00011s latency).

PORT      STATE SERVICE  VERSION
41262/tcp open  ssl/http CherryPy wsgiserver

$ python3 --version
Python 3.7.5

$ apt-cache show python3-cherrypy3
Package: python3-cherrypy3
Source: cherrypy3
Version: 8.9.1-5
<pre>

will test on CentOS7.

Is there anything for the dashboard to change here? I wonder if this is more of an environment/OS-specific issue?

I tried this against current master/octopus (containerized):

root@cpach:~# nmap -sV --version-light --script ssl-poodle -p 8443 10.3.64.20

Starting Nmap 7.60 ( https://nmap.org ) at 2020-01-31 16:53 CST
Stats: 0:00:13 elapsed; 0 hosts completed (1 up), 1 undergoing Service Scan
Service scan Timing: About 0.00% done
Nmap scan report for 10.3.64.20
Host is up (0.000090s latency).

PORT     STATE SERVICE  VERSION
8443/tcp open  ssl/http CherryPy httpd 18.4.0
MAC Address: 0C:C4:7A:D2:C7:70 (Super Micro Computer)

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 13.29 seconds

Sage Weil wrote:

I tried this against current master/octopus (containerized):
[...]
i think we're fine?

Looks so (tested with python3-cherrypy-18.4.0-1.el8.noarch and python3-cheroot-8.2.1-1.el8.noarch) and no trace of SSLv3:

nmap --script ssl-enum-ciphers -p <port> <hostname>
Starting Nmap 7.70 ( https://nmap.org ) at 2020-02-03 10:26 CET
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000099s latency).
Other addresses for localhost (not scanned): ::1

PORT      STATE SERVICE
11013/tcp open  unknown
| ssl-enum-ciphers: 
|   TLSv1.2: 
|     ciphers: 
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (ecdh_x25519) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (ecdh_x25519) - A
|       TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CCM (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CCM (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|     compressors: 
|       NULL
|     cipher preference: server
|_  least strength: A

And example of a positive (vulnerable) output can be found here

Will a fix for this vulernabilty be back ported?

Currently, luminous, mimic, and nautilus are still listed as active.
In particular, we are still using mimic.

REF: [ https://docs.ceph.com/docs/master/releases/general/ ]

A Boschke wrote:

Will a fix for this vulernabilty be back ported?

Currently, luminous, mimic, and nautilus are still listed as active.
In particular, we are still using mimic.

REF: [ https://docs.ceph.com/docs/master/releases/general/ ]

It might be possible to backport it to mimic, but to know for sure I would need the SHA1 (or PR URL) of the fix.

It seems the fix involves building with python3-cherrypy which for EL7 users seems unpackaged as of now.
(Naturally, we're using EL7 :)

A Boschke wrote:

It seems the fix involves building with python3-cherrypy which for EL7 users seems unpackaged as of now.
(Naturally, we're using EL7 :)

Are you sure we would need to build and release new Ceph packages for that, though? Wouldn't it be enough to update your "python2-cherrypy" or "python-cherrypy" package (the one you have installed on the Ceph cluster nodes) to a version containing the fix?

NOTE: Mimic uses Python 2, so "python3-cherrypy" would not help there.

Your question is best answered by developers who did that actual troubleshooting.

A Boschke wrote:

Your question is best answered by developers who did that actual troubleshooting.

ceph-mgr packages are not pinned to any specific CherryPy version, so the only thing you need to do is to install a fixed Cherrpy package from your distro provider. If not available, Cherrypy/Cherroot is a pure-Python package, so you could simply install it from Pypi.

My followup question here really should have been worded differently.

Will the supported Mimic repository's ceph-mgr package be rebuilt so that SSLv3 is disabled?
It's still enabled, yet supported by the project.

If it won't be fixed, we'll the package at least me amended to include a warning about POODLE et al?

A Boschke wrote:

My followup question here really should have been worded differently.

Will the supported Mimic repository's ceph-mgr package be rebuilt so that SSLv3 is disabled?
It's still enabled, yet supported by the project.

If it won't be fixed, we'll the package at least me amended to include a warning about POODLE et al?

I understood what you meant, but when there's a vulnerability in a package the usual practice is to fix the vulnerability in the compromised package and release that very one, not rebuilding or adding warnings to everything else just to work around the compromised piece of code.

In Mimic we were already using Python ssl module, so that moves the focus to Python. In ssl module, since Python 2.7, the default behavior is to reject SSLv3, so unless I'm missing anything here: this would be fixed (in fact no code change has been delivered here).

Are you still experiencing this issue?

Ernesto Puerta wrote:

In Mimic we were already using Python ssl module, so that moves the focus to Python. In ssl module, since Python 2.7, the default behavior is to reject SSLv3, so unless I'm missing anything here: this would be fixed (in fact no code change has been delivered here).

Are you still experiencing this issue?

I'm actually having this on FreeBSD with Nautilus.
But this track leads be to the assumption that Ceph is not at fault here.
Somewhere in the Python stack SSLv3 is not disabled, and that causes the problem.
Which boils down to a FreeBSD packaging configuration thingy.

I'll try the open_ssl test suggested, and track from there

Willem Jan Withagen wrote:

Ernesto Puerta wrote:

In Mimic we were already using Python ssl module, so that moves the focus to Python. In ssl module, since Python 2.7, the default behavior is to reject SSLv3, so unless I'm missing anything here: this would be fixed (in fact no code change has been delivered here).

Are you still experiencing this issue?

I'm actually having this on FreeBSD with Nautilus.
But this track leads be to the assumption that Ceph is not at fault here.
Somewhere in the Python stack SSLv3 is not disabled, and that causes the problem.
Which boils down to a FreeBSD packaging configuration thingy.

I'll try the open_ssl test suggested, and track from there

This is what I get from a connect with openssl ssl3:
CONNECTED
34371092480:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:/usr/src/crypto/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 40
--
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 58 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : SSLv3
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1599390869
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
Exit 1 ========================================

So that does not seem to work.

Giving openssl the free hand, does work and make a TLS 1.3 link.
Now this is on a server where openssl no longer has ssl2/3 compiled in.
So that sort of fixes the problem in a practical way.

But the alternative would be to actually set up the ciphers in the webserver.
Newer CherryPy and likes have provisions for that.

1. openssl s_client connect 192.168.11.201:8081
   CONNECTED
   Can't use SSL_get_servername
   depth=0 O = IT, CN = ceph-dashboard
   verify error:num=18:self signed certificate
   verify return:1
   depth=0 O = IT, CN = ceph-dashboard
   verify return:1
   --
   Certificate chain
   0 s:O = IT, CN = ceph-dashboard
   i:O = IT, CN = ceph-dashboard
   ---
   Server certificate
   -----BEGIN CERTIFICATE-----
   ..............
   -----END CERTIFICATE-----
   subject=O = IT, CN = ceph-dashboard

issuer=O = IT, CN = ceph-dashboard

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1279 bytes and written 375 bytes
Verification error: self signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self signed certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 149A67F50BEBC0F4E089DDEC191C132FC76C0105F05967058F6621EBF4E519A4
Session-ID-ctx:
Resumption PSK: 7C2CCFF4A52D4191B50C542F0C80981A4E6A7C900C61745D0CE3D903B57C0CF6FF7098AAB5B7181C3A4F22863AF3670E
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
...............
Start Time: 1599390927
Timeout : 7200 (sec)
Verify return code: 18 (self signed certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 018BB782E102B854CF3A555C9A08CF3883D9C97753C6B4C97445F034A4428D0F
Session-ID-ctx:
Resumption PSK: BA49EFB8D7A7B2833052841F3E15B3EF1874F6E60FBB800CA2560C3300F1785F14D841E4A4C881676A2FE5AA6BF6FFAA
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
..............
Start Time: 1599390927
Timeout : 7200 (sec)
Verify return code: 18 (self signed certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
HTTP/1.1 408 Request Timeout
Content-Length: 0
Content-Type: text/plain

read:errno=0

Hi Willem, as mentioned we couldn't reproduce this issue in the distros we use so there's no way to validate some of the workarounds discussed, but as you managed to reproduce it, feel free to send a PR with the patch and we'll review that.