Project

General

Profile

Bug #58371

https://telemetry.ceph.com endpoint not available

Added by Jan Horacek about 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Immediate
Assignee:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

hello, it looks like telemetry submission endpoint is not available

```
2022-12-30T08:24:18.412+0000 7f0bd0cbb700 0 [telemetry ERROR root] Failed to send devices to https://telemetry.ceph.com/device: HTTPSConnectionPool(host='telemetry.ceph.com', port=443): Max retries exceeded with url: /device (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0be70e93c8>: Failed to establish a new connection: [Errno 110] Connection timed out',))
```

"ceph telemetry status" says last upload was successful (depending on our clusters) sometime around Wed Dec 28 23:09:02 2022 / Thu Dec 29 02:24:56 2022 (CET timezone)

History

#1 Updated by Yaarit Hatuka about 1 year ago

Hi Jan,

Many thanks for reporting this. We are working on fixing it.

Thanks,
Yaarit

#2 Updated by Yaarit Hatuka about 1 year ago

  • Project changed from Ceph to Infrastructure
  • Priority changed from Normal to Immediate

I restored the DB and the public dashboards (https://telemetry-public.ceph.com/).

Adam Kraitman and Dan Mick are working on restoring the REST endpoint service.

We thank you for your patience, and apologized for any inconvenience.

#3 Updated by Yaarit Hatuka about 1 year ago

  • Status changed from New to Resolved

Hi Jan,

Adam Kraitman and Dan Mick restored the service and now the telemetry endpoints are up and running again.

Please let us know if you experience any other issues.

Thanks for your contribution!
Yaarit

#4 Updated by Oliver Schmidt about 1 year ago

Please let us know if you experience any other issues.

I still have issues connecting to the https://telemetry.ceph.com/report and /device endpoints:

2023-02-02 20:07:59.679 7f0f120d7640 -1 mgr.server reply reply (1) Operation not permitted Failed to send ceph report to https://telemetry.ceph.com/report: HTTPSConnectionPool(host='telemetry.ceph.com', port=443): Max retries exceeded with url: /report (Caused by NewCon
nectionError('<urllib3.connection.HTTPSConnection object at 0x7f0e36abc7c0>: Failed to establish a new connection: [Errno 110] Connection timed out'))
Failed to send devices to https://telemetry.ceph.com/device: HTTPSConnectionPool(host='telemetry.ceph.com', port=443): Max retries exceeded with url: /device (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0e285fabb0>: Failed to establis
h a new connection: [Errno 110] Connection timed out'))

This is still a Nautilus mgr, but as curl shows this seems to be more of an infrastructure issue.

curl -v https://telemetry.ceph.com/report
*   Trying 8.43.84.137:443...
* connect to 8.43.84.137 port 443 failed: Operation timed out
* Failed to connect to telemetry.ceph.com port 443 after 75028 ms: Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to telemetry.ceph.com port 443 after 75028 ms: Couldn't connect to server

#5 Updated by Yaarit Hatuka about 1 year ago

  • Status changed from Resolved to In Progress

Hi Oliver,

Thanks for reporting.
The telemetry endpoints are currently down as we are facing network issues.
We are working on fixing it.
Apologies for the inconvenience.

Yaarit

#6 Updated by Yaarit Hatuka about 1 year ago

  • Status changed from In Progress to Resolved

Hi Oliver,

Network issues are now resolved and telemetry endpoints are up again.

Thanks for your contribution!
Yaarit

Also available in: Atom PDF