Project

General

Profile

Actions

Bug #55638

closed

alertmanager webhook urls may lead to 404

Added by Paul Cuzner almost 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Low
Category:
cephadm/monitoring
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

The current formation of the dashboard urls can result in the url containing a double-slash
When this happens, you see this in the alert manager log

May 12 20:45:20 myserver.ceph.lab conmon[3047303]: level=error ts=2022-05-13T02:45:20.165Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://192.168.1.11:8443//api/prometheus_receiver"  

And this in the mgr log

May 12 20:45:10 occldlr750-1.occl208.lab ceph-mgr[1260392]: [dashboard INFO request] [::ffff:192.168.23.11:49470] [POST] [404] [0.003s] [admin] [513.0B] /prometheus_receiver
                                                            FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/ceph/mgr/dashboard/frontend/dist/en-US/prometheus_receiver'
                                                            cherrypy._cperror.NotFound: (404, "The path '/prometheus_receiver' was not found.")
May 12 20:45:20 occldlr750-1.occl208.lab ceph-mgr[1260392]: [dashboard INFO request] [::ffff:192.168.23.11:49470] [POST] [404] [0.003s] [admin] [513.0B] /prometheus_receiver

The issue lies in the way the dashboard_urls are created, since the template uses a path of /api/prometheus_receiver but a service dump from the mgr includes a trailing '/'

Removing the // from the alertmanager config resolves the issue


Related issues 3 (2 open1 closed)

Related to mgr - Bug #55656: mgr crash on "The path '/prometheus_receiver' was not found."Closed

Actions
Related to Orchestrator - Bug #54469: cephadm/smoke: "Post https://172.21.15.73:8443//api/prometheus_receiver: context deadline exceeded" leads to unresponsive managerNew

Actions
Has duplicate Orchestrator - Bug #58921: Mgr crashing with dashboard module enabled in 16.2.9New

Actions
Actions #1

Updated by Redouane Kachach Elhichou almost 2 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Redouane Kachach Elhichou
  • Pull request ID set to 46306
Actions #2

Updated by Redouane Kachach Elhichou almost 2 years ago

  • Related to Bug #55656: mgr crash on "The path '/prometheus_receiver' was not found." added
Actions #3

Updated by Redouane Kachach Elhichou almost 2 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #4

Updated by Redouane Kachach Elhichou almost 2 years ago

  • Status changed from Pending Backport to Resolved
Actions #5

Updated by Redouane Kachach Elhichou almost 2 years ago

  • Related to Bug #54469: cephadm/smoke: "Post https://172.21.15.73:8443//api/prometheus_receiver: context deadline exceeded" leads to unresponsive manager added
Actions #6

Updated by Ernesto Puerta about 1 year ago

  • Has duplicate Bug #58921: Mgr crashing with dashboard module enabled in 16.2.9 added
Actions

Also available in: Atom PDF