Project

General

Profile

Bug #39628

mgr/dashboard: openssl exception when verifying certificates of HTTPS requests

Added by Ricardo Dias almost 5 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
General - Back-end
Target version:
% Done:

0%

Source:
Tags:
Backport:
nautilus
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When the dashboard backend tries to access an external service provided by a REST API with SSL enabled it fails to verify the connection's SSL certificate with the following stacktrace:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 670, in respond
    response.body = self.handler()
  File "/usr/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 221, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/cherrypy/_cptools.py", line 237, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 60, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/grafana.py", line 37, in validation
    response = grafana.url_validation(method, url)
  File "/usr/share/ceph/mgr/dashboard/controllers/grafana.py", line 16, in url_validation
    path, verify=False)
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
  File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context
    context.verify_mode = cert_reqs
  File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 412, in verify_mode
    _verify_callback
  File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1061, in set_verify
    self._verify_helper = _VerifyHelper(callback)
  File "/usr/lib/python3.6/site-packages/OpenSSL/SSL.py", line 337, in __init__
    "int (*)(int, X509_STORE_CTX *)", wrapper)
SystemError: <built-in method callback of CompiledFFI object at 0x7efe00acf5b8> returned NULL without setting an error


Related issues

Copied to Dashboard - Backport #39962: nautilus: mgr/dashboard: openssl exception when verifying certificates of HTTPS requests Resolved

History

#1 Updated by Ricardo Dias almost 5 years ago

This problem does not happen when running the dashboard in a vstart cluster.

#2 Updated by Ricardo Dias almost 5 years ago

The problem only occurs when executing the ceph-mgr daemon from systemd. The issued is caused by the "MemoryDenyWriteExecute=true" option in ceph-mgr@.service file. This option prevents libraries that generate code dynamically, like JIT compilers, from working as "mmap" calls are filtered by systemd.

In this particular case, my suspicion is that the pyOpenSSL library uses python-cffi library to call the openSSL C library, which probably generates some dynamic code on the fly and fails to do so.

Setting "MemoryDenyWriteExecute=false" fixes the problem.

#3 Updated by Ricardo Dias almost 5 years ago

  • Pull request ID set to 28023

#4 Updated by Kefu Chai almost 5 years ago

  • Status changed from New to Fix Under Review

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=913544 is another occurrence of the same issue

#5 Updated by Lenz Grimmer almost 5 years ago

  • Backport set to nautilus

#6 Updated by Lenz Grimmer almost 5 years ago

  • Subject changed from openssl exception when verifying ceritifcates of HTTPS requests to mgr/dashboard: openssl exception when verifying certificates of HTTPS requests

#7 Updated by Nathan Cutler almost 5 years ago

  • Status changed from Fix Under Review to Pending Backport

#8 Updated by Nathan Cutler almost 5 years ago

  • Copied to Backport #39962: nautilus: mgr/dashboard: openssl exception when verifying certificates of HTTPS requests added

#9 Updated by Lenz Grimmer almost 5 years ago

  • Status changed from Pending Backport to Resolved

#10 Updated by Lenz Grimmer almost 5 years ago

  • Target version set to v15.0.0

#11 Updated by Ernesto Puerta almost 3 years ago

  • Project changed from mgr to Dashboard
  • Category changed from 146 to General - Back-end

Also available in: Atom PDF