Bug #40023: mgr prometheus start failed - mgr - Ceph

Actions

Copy link

Bug #40023

closed

mgr prometheus start failed

Added by yang wang almost 5 years ago. Updated over 4 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Category:

prometheus module

Target version:

% Done:

Source:

Tags:

Backport:

nautilus

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

28246

Crash signature (v1):

Crash signature (v2):

Description

[25/May/2019:13:04:43] ENGINE Bus STOPPING
[25/May/2019:13:04:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down
[25/May/2019:13:04:43] ENGINE Stopped thread '_TimeoutMonitor'.
[25/May/2019:13:04:43] ENGINE Bus STOPPED
2019-05-25 13:04:43.750142 7f3d880de700 1 mgr load Constructed class from module: prometheus
2019-05-25 13:04:43.751225 7f3d880de700 1 mgr load Constructed class from module: restful
2019-05-25 13:04:43.751787 7f3d880de700 1 mgr load Constructed class from module: status
2019-05-25 13:04:43.751858 7f3d880de700 1 mgr send_beacon active
[25/May/2019:13:04:43] ENGINE Bus STARTING
2019-05-25 13:04:43.753823 7f3d8b8e5700 1 mgr[restful] server not running: no certificate configured
[25/May/2019:13:04:43] ENGINE Started monitor thread '_TimeoutMonitor'.
[25/May/2019:13:04:43] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 187, in _start_http_thread
self.httpserver.start()
File "/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1824, in start
raise socket.error(msg)
error: No socket could be created

[25/May/2019:13:04:43] ENGINE Bus STOPPING
[25/May/2019:13:04:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) already shut down
[25/May/2019:13:04:43] ENGINE Stopped thread '_TimeoutMonitor'.
[25/May/2019:13:04:43] ENGINE Bus STOPPED
[25/May/2019:13:04:43] ENGINE Bus EXITING
[25/May/2019:13:04:43] ENGINE Bus EXITED
Exception in thread HTTPServer Thread-16:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 812, in _bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 765, in run
self._target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 187, in _start_http_thread
self.httpserver.start()
File "/usr/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1824, in start
raise socket.error(msg)
error: No socket could be created

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by yang wang almost 5 years ago

Reason:
when mgr is active, it will restart the prometheus 9283 service,
in start process, the cherrypy(i used 3.2.2-4 centos7 defauolt) will check and wait until 9283 was not occupied.
if ipv6 was disabled and server_addr is ::, cherry will use ipv6 lo ::1:9283 to check the port, it always not exist,
so when starting, the previous port may not be stopped, and the new one may start failed with error: No socket could be created

Actions

Copy link