Project

General

Profile

Bug #48973

Updated by Ernesto Puerta about 3 years ago

h3. Description of problem

On first try to access dashboard (typing @https://<dashboard_url>@ and pressing ENTER) the browser (tested in Chrome @Version 87.0.4280.88 (Official Build) (64-bit)@ and Firefox @84.0.1 (64-bit)@) doesn't show any page and it simply keeps waiting.

h3. Environment

* @ceph version@ string: initially verified in master Jan 24th (97480142a69e7ff5bd2abaceb42cffe4b749d00c) but also 1 month ago, Dec 24th (6793756f45f669240de952edd92946541385d090). Reproduced in Pacific.
* Platform (OS/distro/release):
** CentOS 8.3 / Fedora 32
** *NOT NOT REPRODUCED in OpenSUSE Tumbleweed* Tumbleweed
* Cluster details (nodes, monitors, OSDs): minimal vstart cluster with 1 mon + 1 mgr + 3 OSDs. It happens as well in Cephadm deployments.
* Browser used (e.g.: @Version 86.0.4240.198 (Official Build) (64-bit)@):
** Chrome @Version 87.0.4280.88 (Official Build) (64-bit)@
** Firefox @84.0.1 (64-bit)@
* Other:
** *NOT REPRODUCED with plain HTTP (HTTPS is required)*



h3. How reproducible

From a freshly launched dashboard (or an immediately restarted mgr), wait until the initialization finishes (@curl -kv https://<dashboard_url>@ returns the @index.html@). Then switch to a browser (either Chrome or Firefox) and type the dashboard URL in the navigation bar and press ENTER. That's enough to trigger the issue.

Sporadic requests via `curl` don't trigger the issue. It happens when multiple requests are issued at the same time. It can be reproduced from the CLI with Apache benchmark:

<pre><code class="text">
> ab -c20 -n1000 "https://<dashboard_url>/docs"

Benchmarking <dashboard_url> (be patient)
Completed 1000 requests
Completed 2000 requests
SSL handshake failed (5).
Completed 3000 requests
SSL handshake failed (5).
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
...

Complete requests: 10000
Failed requests: 2
(Connect: 0, Receive: 0, Length: 2, Exceptions: 0)
Total transferred: 13387322 bytes
</code></pre>

h3. Actual results

Dashboard login page is not displaying and the browser keeps loading/waiting until manually stopped (minutes). After that, the @curl@ requests no longer work:

<pre><code class="text">
curl -kv https://localhost:11000
* Rebuilt URL to: https://localhost:11000/
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 11000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: O=IT; CN=ceph-dashboard
* start date: Jan 20 16:42:29 2021 GMT
* expire date: Jan 18 16:42:29 2031 GMT
* issuer: O=IT; CN=ceph-dashboard
* SSL certificate verify result: self signed certificate (18), continuing anyway.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET / HTTP/1.1
> Host: localhost:11000
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
</code></pre>

h3. Expected results

Dashboard login page loads normally.

h3. Additional info

Current efforts are led towards finding where this issue comes from:
* Dashboard Python code: code
* Ceph-mgr Python
* Ceph-mgr C++
* CherryPy: reproduced with both @builtin@ and @PyOpenSSL@ transport wrappers. CherryPy

Back