Actions
Bug #38048
openTeuthology error: mgr/prometheus fails with NewConnectionError
Status:
New
Priority:
Normal
Assignee:
-
Category:
prometheus module
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
QA
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2019-01-25T12:09:53.069 INFO:teuthology.orchestra.run.mira069.stderr:2019-01-25 12:09:53.064 7f6415f2b700 1 -- v1:172.21.6.110:46408/2431303140 --> [v2:172.21.6.110:3300/0,v1:172.21.6.110:6789/0] -- mon_command({"prefix": "config set", "who": "mgr", "name": "mgr/prometheus/y/server_port", "value": "7790"} v 0) v1 -- 0x7f641014bfe0 con 0x7f63f40082c0
2019-01-25T12:11:19.581 INFO:tasks.ceph.mgr.y.mira018.stderr:[25/Jan/2019:12:11:19] ENGINE Bus STARTING 2019-01-25T12:11:19.582 INFO:tasks.ceph.mgr.y.mira018.stderr:CherryPy Checker: 2019-01-25T12:11:19.582 INFO:tasks.ceph.mgr.y.mira018.stderr:The Application mounted at '' has an empty config. 2019-01-25T12:11:19.582 INFO:tasks.ceph.mgr.y.mira018.stderr: 2019-01-25T12:11:19.582 INFO:tasks.ceph.mgr.y.mira018.stderr:[25/Jan/2019:12:11:19] ENGINE Started monitor thread '_TimeoutMonitor'. 2019-01-25T12:11:19.691 INFO:tasks.ceph.mgr.y.mira018.stderr:[25/Jan/2019:12:11:19] ENGINE Serving on :::7790 2019-01-25T12:11:19.722 INFO:tasks.ceph.mgr.y.mira018.stderr:[25/Jan/2019:12:11:19] ENGINE Bus STARTED
2019-01-25T12:11:20.252 INFO:tasks.cephfs_test_runner:====================================================================== 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:ERROR: test_standby (tasks.mgr.test_prometheus.TestPrometheus) 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/mgr/test_prometheus.py", line 48, in test_standby 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner: r = requests.get(original_uri, allow_redirects=False) 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 70, in get 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner: return request('get', url, params=params, **kwargs) 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 56, in request 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: return session.request(method=method, url=url, **kwargs) 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: resp = self.send(prep, **send_kwargs) 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: r = adapter.send(request, **kwargs) 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 487, in send 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: raise ConnectionError(e, request=request) 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner:ConnectionError: HTTPConnectionPool(host='mira018.front.sepia.ceph.com', port=7790): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f22994773d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
This was probably a hard to reproduceable timing error
Actions