Project

General

Profile

Bug #20216

mgr: dashboard failure to bind to port 7000 (or whatever) makes mgr exit/fail

Added by Sage Weil about 2 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Immediate
Assignee:
-
Category:
-
Target version:
-
Start date:
06/07/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

2017-06-07 16:49:52.942198 7f4d777fe700 10 mgr notify_all notify_all: notify_all pg_summary
2017-06-07 16:49:52.942203 7f4d777fe700 10 mgr handle_mgr_digest done.
2017-06-07 16:49:52.942241 7f4d52ffd700 20 mgr Gil GIL acquired for thread state 0x7f4d4445c1d0
2017-06-07 16:49:52.942281 7f4d52ffd700 20 mgr ~Gil GIL released for thread state 0x7f4d4445c1d0
2017-06-07 16:49:52.942294 7f4d52ffd700 20 mgr Gil GIL acquired for thread state 0x7f4d4445c1d0
2017-06-07 16:49:52.942300 7f4d52ffd700 20 mgr ~Gil GIL released for thread state 0x7f4d4445c1d0
2017-06-07 16:49:52.942304 7f4d52ffd700 20 mgr Gil GIL acquired for thread state 0x7f4d4445c1d0
2017-06-07 16:49:52.942548 7f4d52ffd700 20 mgr ~Gil GIL released for thread state 0x7f4d4445c1d0
2017-06-07 16:49:54.017312 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Error in 'start' listener <bound method Server.start of <cherrypy._cpserver.Server object at 0x7f4d2e102850>>
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 205, in publish
    output.append(listener(*args, **kwargs))
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpserver.py", line 168, in start
    ServerAdapter.start(self)
  File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 170, in start
    wait_for_free_port(*self.bind_addr)
  File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 438, in wait_for_free_port
    raise IOError("Port %r not free on %r" % (port, host))
IOError: Port 7000 not free on '127.0.0.1'

2017-06-07 16:49:54.017681 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Shutting down due to error in start listener:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 243, in start
    self.publish('start')
  File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 223, in publish
    raise exc
ChannelFailures: IOError("Port 7000 not free on '127.0.0.1'",)

2017-06-07 16:49:54.017879 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus STOPPING
2017-06-07 16:49:54.018112 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 7000)) already shut down
2017-06-07 16:49:54.018345 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Stopped thread '_TimeoutMonitor'.
2017-06-07 16:49:54.018532 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus STOPPED
2017-06-07 16:49:54.018704 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus EXITING
2017-06-07 16:49:54.018881 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus EXITED

and the process seems to exit.

reproducible on master if someone else is bound to port 7000 (or whatever)

History

#1 Updated by Sage Weil about 2 months ago

  • Status changed from New to Verified

This appears to be something with cherrypy.. when restful fails to bind the module doesn't start but mgr survives. not so with dashboard.

#2 Updated by Tim Serong about 2 months ago

I'll try to do something about this. Looks like cherrypy will call os_.exit() if exceptions are thrown during startup or shutdown, which is a bit... irritating.

#3 Updated by Tim Serong about 2 months ago

Adding the following to src/pybind/mgr/dashboard/module.py "fixes" the problem.

def os_exit_noop():
    pass

os._exit = os_exit_noop

Not sure if this is a terrible idea or not. Thoughts, anyone?

#4 Updated by John Spray about 2 months ago

Seriously antisocial behaviour from cherrypy there. In this context monkey patching os._exit doesn't seem unreasonable to me.

#5 Updated by Sage Weil about 2 months ago

  • Status changed from Verified to Need Review

#6 Updated by Sage Weil about 2 months ago

  • Status changed from Need Review to Resolved

Also available in: Atom PDF