Bug #48973
Updated by Ernesto Puerta about 3 years ago
h3. Description of problem On first try to access dashboard (typing @https://<dashboard_url>@ and pressing ENTER) the browser (tested in Chrome @Version 87.0.4280.88 (Official Build) (64-bit)@ and Firefox @84.0.1 (64-bit)@) doesn't show any page and it simply keeps waiting. h3. Environment * @ceph version@ string: initially verified in master Jan 24th (97480142a69e7ff5bd2abaceb42cffe4b749d00c) but also 1 month ago, Dec 24th (6793756f45f669240de952edd92946541385d090). Reproduced in Pacific. * Platform (OS/distro/release): ** CentOS 8.3 / Fedora 32 ** NOT REPRODUCED in OpenSUSE Tumbleweed * Cluster details (nodes, monitors, OSDs): minimal vstart cluster with 1 mon + 1 mgr + 3 OSDs. It happens as well in Cephadm deployments. * Browser used (e.g.: @Version 86.0.4240.198 (Official Build) (64-bit)@): ** Chrome @Version 87.0.4280.88 (Official Build) (64-bit)@ ** Firefox @84.0.1 (64-bit)@ h3. How reproducible From a freshly launched dashboard (or an immediately restarted mgr), wait until the initialization finishes (@curl -kv https://<dashboard_url>@ returns the @index.html@). Then switch to a browser (either Chrome or Firefox) and type the dashboard URL in the navigation bar and press ENTER. That's enough to trigger the issue. Sporadic requests via `curl` don't trigger the issue. It happens when multiple requests are issued at the same time. It can be reproduced from the CLI with Apache benchmark: <pre><code class="text"> ab -c20 -n1000 -H "accept: /" -H "Authorization: Bearer $AB_AUTH" "https://172.20.0.2:11000/docs" </code></pre> h3. Actual results Dashboard login page is not displaying and the browser keeps loading/waiting until manually stopped (minutes). After that, the @curl@ requests no longer work: <pre><code class="text"> curl -kv https://localhost:11000 * Rebuilt URL to: https://localhost:11000/ * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 11000 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, [no content] (0): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: O=IT; CN=ceph-dashboard * start date: Jan 20 16:42:29 2021 GMT * expire date: Jan 18 16:42:29 2031 GMT * issuer: O=IT; CN=ceph-dashboard * SSL certificate verify result: self signed certificate (18), continuing anyway. * TLSv1.3 (OUT), TLS app data, [no content] (0): > GET / HTTP/1.1 > Host: localhost:11000 > User-Agent: curl/7.61.1 > Accept: */* > * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): </code></pre> h3. Expected results Dashboard login page loads normally. h3. Additional info Current efforts are led towards finding where this issue comes from: * Dashboard Python code * Ceph-mgr Python * Ceph-mgr C++ * CherryPy