Actions
Bug #20216
closedmgr: dashboard failure to bind to port 7000 (or whatever) makes mgr exit/fail
Status:
Resolved
Priority:
Immediate
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2017-06-07 16:49:52.942198 7f4d777fe700 10 mgr notify_all notify_all: notify_all pg_summary 2017-06-07 16:49:52.942203 7f4d777fe700 10 mgr handle_mgr_digest done. 2017-06-07 16:49:52.942241 7f4d52ffd700 20 mgr Gil GIL acquired for thread state 0x7f4d4445c1d0 2017-06-07 16:49:52.942281 7f4d52ffd700 20 mgr ~Gil GIL released for thread state 0x7f4d4445c1d0 2017-06-07 16:49:52.942294 7f4d52ffd700 20 mgr Gil GIL acquired for thread state 0x7f4d4445c1d0 2017-06-07 16:49:52.942300 7f4d52ffd700 20 mgr ~Gil GIL released for thread state 0x7f4d4445c1d0 2017-06-07 16:49:52.942304 7f4d52ffd700 20 mgr Gil GIL acquired for thread state 0x7f4d4445c1d0 2017-06-07 16:49:52.942548 7f4d52ffd700 20 mgr ~Gil GIL released for thread state 0x7f4d4445c1d0 2017-06-07 16:49:54.017312 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Error in 'start' listener <bound method Server.start of <cherrypy._cpserver.Server object at 0x7f4d2e102850>> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 205, in publish output.append(listener(*args, **kwargs)) File "/usr/lib/python2.7/site-packages/cherrypy/_cpserver.py", line 168, in start ServerAdapter.start(self) File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 170, in start wait_for_free_port(*self.bind_addr) File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 438, in wait_for_free_port raise IOError("Port %r not free on %r" % (port, host)) IOError: Port 7000 not free on '127.0.0.1' 2017-06-07 16:49:54.017681 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Shutting down due to error in start listener: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 243, in start self.publish('start') File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 223, in publish raise exc ChannelFailures: IOError("Port 7000 not free on '127.0.0.1'",) 2017-06-07 16:49:54.017879 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus STOPPING 2017-06-07 16:49:54.018112 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 7000)) already shut down 2017-06-07 16:49:54.018345 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Stopped thread '_TimeoutMonitor'. 2017-06-07 16:49:54.018532 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus STOPPED 2017-06-07 16:49:54.018704 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus EXITING 2017-06-07 16:49:54.018881 7f4d23753700 4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus EXITED
and the process seems to exit.
reproducible on master if someone else is bound to port 7000 (or whatever)
Updated by Sage Weil almost 7 years ago
- Status changed from New to 12
This appears to be something with cherrypy.. when restful fails to bind the module doesn't start but mgr survives. not so with dashboard.
Updated by Tim Serong almost 7 years ago
I'll try to do something about this. Looks like cherrypy will call os_.exit() if exceptions are thrown during startup or shutdown, which is a bit... irritating.
Updated by Tim Serong almost 7 years ago
Adding the following to src/pybind/mgr/dashboard/module.py "fixes" the problem.
def os_exit_noop(): pass os._exit = os_exit_noop
Not sure if this is a terrible idea or not. Thoughts, anyone?
Updated by John Spray almost 7 years ago
Seriously antisocial behaviour from cherrypy there. In this context monkey patching os._exit doesn't seem unreasonable to me.
Updated by Sage Weil almost 7 years ago
- Status changed from 12 to Fix Under Review
Updated by Sage Weil almost 7 years ago
- Status changed from Fix Under Review to Resolved
Updated by Yuri Weinstein about 6 years ago
- Related to Bug #23483: "ENGINE Error in 'start' listener <bound " in rados added
Updated by Abhishek Lekshmanan about 6 years ago
- Status changed from Resolved to Pending Backport
- Backport set to luminous
Updated by Abhishek Lekshmanan about 6 years ago
- Copied to Backport #23484: luminous: mgr: dashboard failure to bind to port 7000 (or whatever) makes mgr exit/fail added
Updated by Abhishek Lekshmanan about 6 years ago
- Status changed from Pending Backport to Closed
Updated by Abhishek Lekshmanan about 6 years ago
- Status changed from Closed to Resolved
Actions