Project

General

Profile

Bug #40023

mgr prometheus start failed

Added by yang wang 4 months ago. Updated 24 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
prometheus module
Target version:
-
Start date:
05/27/2019
Due date:
% Done:

0%

Source:
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

[25/May/2019:13:04:43] ENGINE Bus STOPPING
[25/May/2019:13:04:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down
[25/May/2019:13:04:43] ENGINE Stopped thread '_TimeoutMonitor'.
[25/May/2019:13:04:43] ENGINE Bus STOPPED
2019-05-25 13:04:43.750142 7f3d880de700 1 mgr load Constructed class from module: prometheus
2019-05-25 13:04:43.751225 7f3d880de700 1 mgr load Constructed class from module: restful
2019-05-25 13:04:43.751787 7f3d880de700 1 mgr load Constructed class from module: status
2019-05-25 13:04:43.751858 7f3d880de700 1 mgr send_beacon active
[25/May/2019:13:04:43] ENGINE Bus STARTING
2019-05-25 13:04:43.753823 7f3d8b8e5700 1 mgr[restful] server not running: no certificate configured
[25/May/2019:13:04:43] ENGINE Started monitor thread '_TimeoutMonitor'.
[25/May/2019:13:04:43] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 187, in _start_http_thread
self.httpserver.start()
File "/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1824, in start
raise socket.error(msg)
error: No socket could be created

[25/May/2019:13:04:43] ENGINE Bus STOPPING
[25/May/2019:13:04:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) already shut down
[25/May/2019:13:04:43] ENGINE Stopped thread '_TimeoutMonitor'.
[25/May/2019:13:04:43] ENGINE Bus STOPPED
[25/May/2019:13:04:43] ENGINE Bus EXITING
[25/May/2019:13:04:43] ENGINE Bus EXITED
Exception in thread HTTPServer Thread-16:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 812, in _bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 765, in run
self.
_target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 187, in _start_http_thread
self.httpserver.start()
File "/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1824, in start
raise socket.error(msg)
error: No socket could be created


Related issues

Copied to mgr - Backport #40273: nautilus: mgr prometheus start failed Resolved

History

#1 Updated by yang wang 4 months ago

Reason:
when mgr is active, it will restart the prometheus 9283 service,
in start process, the cherrypy(i used 3.2.2-4 centos7 defauolt) will check and wait until 9283 was not occupied.
if ipv6 was disabled and server_addr is ::, cherry will use ipv6 lo ::1:9283 to check the port, it always not exist,
so when starting, the previous port may not be stopped, and the new one may start failed with error: No socket could be created

#2 Updated by Sebastian Wagner 4 months ago

  • Category set to prometheus module
  • Backport set to nautilus

#3 Updated by Sebastian Wagner 4 months ago

  • Status changed from New to In Progress
  • Pull request ID set to 28246

#4 Updated by Kefu Chai 3 months ago

  • Status changed from In Progress to Pending Backport

#5 Updated by Nathan Cutler 3 months ago

#6 Updated by Nathan Cutler 24 days ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF