Project

General

Profile

Bug #55638

alertmanager webhook urls may lead to 404

Added by Paul Cuzner 7 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Low
Category:
cephadm/monitoring
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

The current formation of the dashboard urls can result in the url containing a double-slash
When this happens, you see this in the alert manager log

May 12 20:45:20 myserver.ceph.lab conmon[3047303]: level=error ts=2022-05-13T02:45:20.165Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: https://192.168.1.11:8443//api/prometheus_receiver"  

And this in the mgr log

May 12 20:45:10 occldlr750-1.occl208.lab ceph-mgr[1260392]: [dashboard INFO request] [::ffff:192.168.23.11:49470] [POST] [404] [0.003s] [admin] [513.0B] /prometheus_receiver
                                                            FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/ceph/mgr/dashboard/frontend/dist/en-US/prometheus_receiver'
                                                            cherrypy._cperror.NotFound: (404, "The path '/prometheus_receiver' was not found.")
May 12 20:45:20 occldlr750-1.occl208.lab ceph-mgr[1260392]: [dashboard INFO request] [::ffff:192.168.23.11:49470] [POST] [404] [0.003s] [admin] [513.0B] /prometheus_receiver

The issue lies in the way the dashboard_urls are created, since the template uses a path of /api/prometheus_receiver but a service dump from the mgr includes a trailing '/'

Removing the // from the alertmanager config resolves the issue


Related issues

Related to mgr - Bug #55656: mgr crash on "The path '/prometheus_receiver' was not found." Closed
Related to Orchestrator - Bug #54469: cephadm/smoke: "Post https://172.21.15.73:8443//api/prometheus_receiver: context deadline exceeded" leads to unresponsive manager New

History

#1 Updated by Redouane Kachach Elhichou 7 months ago

  • Status changed from New to Fix Under Review
  • Assignee set to Redouane Kachach Elhichou
  • Pull request ID set to 46306

#2 Updated by Redouane Kachach Elhichou 7 months ago

  • Related to Bug #55656: mgr crash on "The path '/prometheus_receiver' was not found." added

#3 Updated by Redouane Kachach Elhichou 7 months ago

  • Status changed from Fix Under Review to Pending Backport

#4 Updated by Redouane Kachach Elhichou 7 months ago

  • Status changed from Pending Backport to Resolved

#5 Updated by Redouane Kachach Elhichou 7 months ago

  • Related to Bug #54469: cephadm/smoke: "Post https://172.21.15.73:8443//api/prometheus_receiver: context deadline exceeded" leads to unresponsive manager added

Also available in: Atom PDF