Project

General

Profile

Bug #20216

mgr: dashboard failure to bind to port 7000 (or whatever) makes mgr exit/fail

Added by Sage Weil almost 7 years ago. Updated about 6 years ago.

Status:
Resolved
Priority:
Immediate
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2017-06-07 16:49:52.942198 7f4d777fe700 10 mgr notify_all notify_all: notify_all pg_summary
2017-06-07 16:49:52.942203 7f4d777fe700 10 mgr handle_mgr_digest done.
2017-06-07 16:49:52.942241 7f4d52ffd700 20 mgr Gil GIL acquired for thread state 0x7f4d4445c1d0
2017-06-07 16:49:52.942281 7f4d52ffd700 20 mgr ~Gil GIL released for thread state 0x7f4d4445c1d0
2017-06-07 16:49:52.942294 7f4d52ffd700 20 mgr Gil GIL acquired for thread state 0x7f4d4445c1d0
2017-06-07 16:49:52.942300 7f4d52ffd700 20 mgr ~Gil GIL released for thread state 0x7f4d4445c1d0
2017-06-07 16:49:52.942304 7f4d52ffd700 20 mgr Gil GIL acquired for thread state 0x7f4d4445c1d0
2017-06-07 16:49:52.942548 7f4d52ffd700 20 mgr ~Gil GIL released for thread state 0x7f4d4445c1d0
2017-06-07 16:49:54.017312 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Error in 'start' listener <bound method Server.start of <cherrypy._cpserver.Server object at 0x7f4d2e102850>>
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 205, in publish
    output.append(listener(*args, **kwargs))
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpserver.py", line 168, in start
    ServerAdapter.start(self)
  File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 170, in start
    wait_for_free_port(*self.bind_addr)
  File "/usr/lib/python2.7/site-packages/cherrypy/process/servers.py", line 438, in wait_for_free_port
    raise IOError("Port %r not free on %r" % (port, host))
IOError: Port 7000 not free on '127.0.0.1'

2017-06-07 16:49:54.017681 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Shutting down due to error in start listener:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 243, in start
    self.publish('start')
  File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 223, in publish
    raise exc
ChannelFailures: IOError("Port 7000 not free on '127.0.0.1'",)

2017-06-07 16:49:54.017879 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus STOPPING
2017-06-07 16:49:54.018112 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('127.0.0.1', 7000)) already shut down
2017-06-07 16:49:54.018345 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Stopped thread '_TimeoutMonitor'.
2017-06-07 16:49:54.018532 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus STOPPED
2017-06-07 16:49:54.018704 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus EXITING
2017-06-07 16:49:54.018881 7f4d23753700  4 mgr[py] [07/Jun/2017:16:49:54] ENGINE Bus EXITED

and the process seems to exit.

reproducible on master if someone else is bound to port 7000 (or whatever)


Related issues

Related to Ceph - Bug #23483: "ENGINE Error in 'start' listener <bound " in rados Resolved 03/28/2018
Copied to Ceph - Backport #23484: luminous: mgr: dashboard failure to bind to port 7000 (or whatever) makes mgr exit/fail Closed

History

#1 Updated by Sage Weil almost 7 years ago

  • Status changed from New to 12

This appears to be something with cherrypy.. when restful fails to bind the module doesn't start but mgr survives. not so with dashboard.

#2 Updated by Tim Serong almost 7 years ago

I'll try to do something about this. Looks like cherrypy will call os_.exit() if exceptions are thrown during startup or shutdown, which is a bit... irritating.

#3 Updated by Tim Serong almost 7 years ago

Adding the following to src/pybind/mgr/dashboard/module.py "fixes" the problem.

def os_exit_noop():
    pass

os._exit = os_exit_noop

Not sure if this is a terrible idea or not. Thoughts, anyone?

#4 Updated by John Spray almost 7 years ago

Seriously antisocial behaviour from cherrypy there. In this context monkey patching os._exit doesn't seem unreasonable to me.

#5 Updated by Sage Weil almost 7 years ago

  • Status changed from 12 to Fix Under Review

#6 Updated by Sage Weil almost 7 years ago

  • Status changed from Fix Under Review to Resolved

#7 Updated by Yuri Weinstein about 6 years ago

  • Related to Bug #23483: "ENGINE Error in 'start' listener <bound " in rados added

#8 Updated by Abhishek Lekshmanan about 6 years ago

  • Status changed from Resolved to Pending Backport
  • Backport set to luminous

#9 Updated by Abhishek Lekshmanan about 6 years ago

  • Copied to Backport #23484: luminous: mgr: dashboard failure to bind to port 7000 (or whatever) makes mgr exit/fail added

#10 Updated by Abhishek Lekshmanan about 6 years ago

  • Status changed from Pending Backport to Closed

#11 Updated by Abhishek Lekshmanan about 6 years ago

  • Status changed from Closed to Resolved

Also available in: Atom PDF