Project

General

Profile

Bug #39628

mgr/dashboard: openssl exception when verifying certificates of HTTPS requests

Added by Ricardo Dias 4 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
dashboard/backend
Target version:
Start date:
05/08/2019
Due date:
% Done:

0%

Source:
Tags:
Backport:
nautilus
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

When the dashboard backend tries to access an external service provided by a REST API with SSL enabled it fails to verify the connection's SSL certificate with the following stacktrace:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 670, in respond
    response.body = self.handler()
  File "/usr/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 221, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/cherrypy/_cptools.py", line 237, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 60, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/grafana.py", line 37, in validation
    response = grafana.url_validation(method, url)
  File "/usr/share/ceph/mgr/dashboard/controllers/grafana.py", line 16, in url_validation
    path, verify=False)
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
  File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
    context.verify_mode = cert_reqs
  File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 412, in verify_mode
    _verify_callback
  File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1061, in set_verify
    self._verify_helper = _VerifyHelper(callback)
  File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 337, in __init__
    "int (*)(int, X509_STORE_CTX *)", wrapper)
SystemError: <built-in method callback of CompiledFFI object at 0x7efe00acf5b8> returned NULL without setting an error


Related issues

Copied to mgr - Backport #39962: nautilus: mgr/dashboard: openssl exception when verifying certificates of HTTPS requests Resolved

History

#1 Updated by Ricardo Dias 4 months ago

This problem does not happen when running the dashboard in a vstart cluster.

#2 Updated by Ricardo Dias 4 months ago

The problem only occurs when executing the ceph-mgr daemon from systemd. The issued is caused by the "MemoryDenyWriteExecute=true" option in ceph-mgr@.service file. This option prevents libraries that generate code dynamically, like JIT compilers, from working as "mmap" calls are filtered by systemd.

In this particular case, my suspicion is that the pyOpenSSL library uses python-cffi library to call the openSSL C library, which probably generates some dynamic code on the fly and fails to do so.

Setting "MemoryDenyWriteExecute=false" fixes the problem.

#3 Updated by Ricardo Dias 4 months ago

  • Pull request ID set to 28023

#4 Updated by Kefu Chai 4 months ago

  • Status changed from New to Need Review

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=913544 is another occurrence of the same issue

#5 Updated by Lenz Grimmer 4 months ago

  • Backport set to nautilus

#6 Updated by Lenz Grimmer 3 months ago

  • Subject changed from openssl exception when verifying ceritifcates of HTTPS requests to mgr/dashboard: openssl exception when verifying certificates of HTTPS requests

#7 Updated by Nathan Cutler 3 months ago

  • Status changed from Need Review to Pending Backport

#8 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #39962: nautilus: mgr/dashboard: openssl exception when verifying certificates of HTTPS requests added

#9 Updated by Lenz Grimmer 3 months ago

  • Status changed from Pending Backport to Resolved

#10 Updated by Lenz Grimmer 3 months ago

  • Target version set to v15.0.0

Also available in: Atom PDF