Bug #54469: cephadm/smoke: "Post https://172.21.15.73:8443//api/prometheus_receiver: context deadline exceeded" leads to unresponsive manager - Orchestrator - Ceph

Actions

Copy link

Bug #54469

open

cephadm/smoke: "Post https://172.21.15.73:8443//api/prometheus_receiver: context deadline exceeded" leads to unresponsive manager

Added by Laura Flores about 2 years ago. Updated about 2 years ago.

Status:

New

Priority:

Normal

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

/a/yuriw-2022-03-01_17:45:51-rados-wip-yuri3-testing-2022-02-28-0757-pacific-distro-default-smithi/6714644

2022-03-01T19:29:44.519 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 01 19:29:44 smithi183 conmon[56929]: t=2022-03-01T19:29:44+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml
2022-03-01T19:29:44.519 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 01 19:29:44 smithi183 conmon[56929]: t=2022-03-01T19:29:44+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1
2022-03-01T19:29:44.519 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 01 19:29:44 smithi183 conmon[56929]: t=2022-03-01T19:29:44+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS
2022-03-01T19:29:44.520 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 01 19:29:44 smithi183 conmon[56929]: t=2022-03-01T19:29:44+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS
2022-03-01T19:29:44.520 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 01 19:29:44 smithi183 conmon[56929]: t=2022-03-01T19:29:44+0000 lvl=info msg="Initializing Stream Manager" 
2022-03-01T19:29:44.520 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 01 19:29:44 smithi183 conmon[56929]: t=2022-03-01T19:29:44+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket=
2022-03-01T19:29:54.002 INFO:journalctl@ceph.alertmanager.a.smithi073.stdout:Mar 01 19:29:53 smithi073 conmon[61041]: level=error ts=2022-03-01T19:29:53.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.183:8443/api/prometheus_receiver: context deadline exceeded" context_err="context deadline exceeded" 
2022-03-01T19:29:54.002 INFO:journalctl@ceph.alertmanager.a.smithi073.stdout:Mar 01 19:29:53 smithi073 conmon[61041]: level=error ts=2022-03-01T19:29:53.505Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="Post https://172.21.15.73:8443//api/prometheus_receiver: context deadline exceeded" context_err="context deadline exceeded" 
2022-03-01T19:29:54.003 INFO:journalctl@ceph.alertmanager.a.smithi073.stdout:Mar 01 19:29:53 smithi073 conmon[61041]: level=error ts=2022-03-01T19:29:53.505Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="Post https://172.21.15.183:8443/api/prometheus_receiver: context deadline exceeded; Post https://172.21.15.73:8443//api/prometheus_receiver: context deadline exceeded" 
2022-03-01T19:29:59.252 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[29248]: cluster 2022-03-01T19:29:58.783641+0000 mon.a (mon
2022-03-01T19:29:59.252 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[29248]: .0) 679 : cluster [INF] Manager daemon x is unresponsive.  No standby daemons available.
2022-03-01T19:29:59.253 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[29248]: cluster 2022-03-01T19:29:58.783861+0000 mon.a (mon.0) 680 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)
2022-03-01T19:29:59.253 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[29248]: cluster 2022-03-01T19:29:58.786316+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in
2022-03-01T19:29:59.253 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[29248]: cluster 2022-03-01T19:29:58.791032+0000 mon.a (mon.0) 682 : cluster [DBG] mgrmap e23: no daemons active (since 0.00739813s)
2022-03-01T19:29:59.253 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[36869]: cluster 2022-03-01T19:29:58.783641+0000 mon.a (
2022-03-01T19:29:59.254 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[36869]: mon.0) 679 : cluster [INF] Manager daemon x is unresponsive.  No standby daemons available.
2022-03-01T19:29:59.254 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[36869]: cluster 2022-03-01T19:29:58.783861+0000 mon.a (mon.0) 680 :
2022-03-01T19:29:59.254 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[36869]: cluster [WRN] Health check failed: no active mgr (MGR_DOWN)
2022-03-01T19:29:59.254 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[36869]: cluster 2022-03-01
2022-03-01T19:29:59.255 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[36869]: T19:29:58.786316+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in
2022-03-01T19:29:59.255 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[36869]: cluster 2022-03-
2022-03-01T19:29:59.255 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[36869]: 01T19:29:58.791032+0000 mon.a (mon.0)
2022-03-01T19:29:59.255 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:29:58 smithi073 conmon[36869]: 682 : cluster [DBG] mgrmap e23: no daemons active (since 0.00739813s)
2022-03-01T19:29:59.264 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:29:58 smithi183 conmon[30018]: cluster 2022-03-01T19:29:58.
2022-03-01T19:29:59.264 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:29:58 smithi183 conmon[30018]: 783641+0000 mon.a (mon.0) 679 : cluster [INF] Manager daemon x is unresponsive.  No standby daemons available.
2022-03-01T19:29:59.264 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:29:58 smithi183 conmon[30018]: cluster 2022-03-01T19:29:58.783861+0000 mon.a
2022-03-01T19:29:59.264 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:29:58 smithi183 conmon[30018]:  (mon.0) 680 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)
2022-03-01T19:29:59.265 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:29:58 smithi183 conmon[30018]: cluster 2022-03-01T
2022-03-01T19:29:59.265 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:29:58 smithi183 conmon[30018]: 19:29:58.786316+0000 mon.a (mon.
2022-03-01T19:29:59.265 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:29:58 smithi183 conmon[30018]: 0) 681 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in
2022-03-01T19:29:59.265 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:29:58 smithi183 conmon[30018]:
2022-03-01T19:29:59.266 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:29:58 smithi183 conmon[30018]: cluster 2022-03-01T19:29:58
2022-03-01T19:29:59.266 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:29:58 smithi183 conmon[30018]: .791032+0000 mon.a (mon.0) 682
2022-03-01T19:29:59.266 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:29:58 smithi183 conmon[30018]:  : cluster [DBG] mgrmap e23: no daemons active (since 0.00739813s)
2022-03-01T19:30:00.502 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 01 19:30:00 smithi073 conmon[29248]: cluster 2022-03-01T19:30:00.
2022-03-01T19:30:00.502 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 01 19:30:00 smithi073 conmon[29248]: 000142+0000 mon.a (mon.0) 683 : cluster [WRN] Health detail: HEALTH_WARN no active mgr
2022-03-01T19:30:00.502 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 01 19:30:00 smithi073 conmon[29248]: cluster 2022-03-01T19:
2022-03-01T19:30:00.503 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 01 19:30:00 smithi073 conmon[29248]: 30:00.000162+0000 mon.a (mon.0) 684 : cluster [WRN] [WRN] MGR_DOWN: no active mgr
2022-03-01T19:30:00.503 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:30:00 smithi073 conmon[36869]: cluster 2022-03-01T19:30:00.000142+0000 mon.a
2022-03-01T19:30:00.503 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:30:00 smithi073 conmon[36869]:  (mon.0) 683 : cluster [WRN] Health detail: HEALTH_WARN no active mgr
2022-03-01T19:30:00.504 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:30:00 smithi073 conmon[36869]: cluster 2022-03-
2022-03-01T19:30:00.504 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:30:00 smithi073 conmon[36869]: 01T19:30:00.000162+0000 mon.a (mon.0) 684 : cluster
2022-03-01T19:30:00.504 INFO:journalctl@ceph.mon.c.smithi073.stdout:Mar 01 19:30:00 smithi073 conmon[36869]:  [WRN] [WRN] MGR_DOWN: no active mgr
2022-03-01T19:30:00.513 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:30:00 smithi183 conmon[30018]: cluster 2022-03-01T19:30:00.000142
2022-03-01T19:30:00.513 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:30:00 smithi183 conmon[30018]: +0000 mon.a (mon.0) 683 : cluster [WRN] Health detail: HEALTH_WARN no active mgr
2022-03-01T19:30:00.514 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:30:00 smithi183 conmon[30018]: cluster 2022-03-01T19:30:00.000162+0000 mon.a
2022-03-01T19:30:00.514 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 01 19:30:00 smithi183 conmon[30018]:  (mon.0) 684 : cluster [WRN] [WRN] MGR_DOWN: no active mgr

Related issues 1 (0 open — 1 closed)