Bug #20216
mgr: dashboard failure to bind to port 7000 (or whatever) makes mgr exit/fail
Status:
Resolved
Priority:
Immediate
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2017-06-07 16:49:52.942198 7f4d777fe700 10 mgr notify_all notify_all: notify_all pg_summary 2017-06-07 16:49:52.942203 7f4d777fe700 10 mgr handle_mgr_digest done. 2017-06-07 16:49:52.942241 7f4d52ffd700 20 mgr Gil GIL acquired for thread state 0x7f4d4445c1d0 2017-06-07 16:49:52.942281 7f4d52ffd700 20 mgr ~Gil GIL released for thread state 0x7f4d4445c1d0 2017-06-07 16:49:52.942294 7f4d52ffd700 20 mgr Gil GIL acquired for thread state 0x7f4d4445c1d0 2017-06-07 16:49:52.942300 7f4d52ffd700 20 mgr ~Gil GIL released for thread state 0x7f4d4445c1d0 2017-06-07 16:49:52.942304 7f4d52ffd700 20 mgr Gil GIL acquired for thread state 0x7f4d4445c1d0 2017-06-07 16:49:52.942548 7f4d52ffd700 20 mgr ~Gil GIL released for thread state 0x7f4d4445c1d0 2017-06-07 16:49:54.017312 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Error in 'start' listener <bound method Server.start of <cherrypy._cpserver.Server object at 0x7f4d2e102850>> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 205, in publish output.append(listener(*args, **kwargs)) File "/usr/lib/python2.7/site-packages/cherrypy/_cpserver.py", line 168, in start ServerAdapter.start(self) File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 170, in start wait_for_free_port(*self.bind_addr) File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 438, in wait_for_free_port raise IOError("Port %r not free on %r" % (port, host)) IOError: Port 7000 not free on '127.0.0.1' 2017-06-07 16:49:54.017681 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Shutting down due to error in start listener: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 243, in start self.publish('start') File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 223, in publish raise exc ChannelFailures: IOError("Port 7000 not free on '127.0.0.1'",) 2017-06-07 16:49:54.017879 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus STOPPING 2017-06-07 16:49:54.018112 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 7000)) already shut down 2017-06-07 16:49:54.018345 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Stopped thread '_TimeoutMonitor'. 2017-06-07 16:49:54.018532 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus STOPPED 2017-06-07 16:49:54.018704 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus EXITING 2017-06-07 16:49:54.018881 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus EXITED
and the process seems to exit.
reproducible on master if someone else is bound to port 7000 (or whatever)
Related issues
History
#1 Updated by Sage Weil almost 7 years ago
- Status changed from New to 12
This appears to be something with cherrypy.. when restful fails to bind the module doesn't start but mgr survives. not so with dashboard.
#2 Updated by Tim Serong almost 7 years ago
I'll try to do something about this. Looks like cherrypy will call os_.exit() if exceptions are thrown during startup or shutdown, which is a bit... irritating.
#3 Updated by Tim Serong almost 7 years ago
Adding the following to src/pybind/mgr/dashboard/module.py "fixes" the problem.
def os_exit_noop(): pass os._exit = os_exit_noop
Not sure if this is a terrible idea or not. Thoughts, anyone?
#4 Updated by John Spray almost 7 years ago
Seriously antisocial behaviour from cherrypy there. In this context monkey patching os._exit doesn't seem unreasonable to me.
#5 Updated by Sage Weil almost 7 years ago
- Status changed from 12 to Fix Under Review
#6 Updated by Sage Weil almost 7 years ago
- Status changed from Fix Under Review to Resolved
#7 Updated by Yuri Weinstein about 6 years ago
- Related to Bug #23483: "ENGINE Error in 'start' listener <bound " in rados added
#8 Updated by Abhishek Lekshmanan about 6 years ago
- Status changed from Resolved to Pending Backport
- Backport set to luminous
#9 Updated by Abhishek Lekshmanan about 6 years ago
- Copied to Backport #23484: luminous: mgr: dashboard failure to bind to port 7000 (or whatever) makes mgr exit/fail added
#10 Updated by Abhishek Lekshmanan about 6 years ago
- Status changed from Pending Backport to Closed
#11 Updated by Abhishek Lekshmanan about 6 years ago
- Status changed from Closed to Resolved