Bug #38048
Teuthology error: mgr/prometheus fails with NewConnectionError
Status:
New
Priority:
Normal
Assignee:
-
Category:
prometheus module
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
QA
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2019-01-25T12:09:53.069 INFO:teuthology.orchestra.run.mira069.stderr:2019-01-25 12:09:53.064 7f6415f2b700 1 -- v1:172.21.6.110:46408/2431303140 --> [v2:172.21.6.110:3300/0,v1:172.21.6.110:6789/0] -- mon_command({"prefix": "config set", "who": "mgr", "name": "mgr/prometheus/y/server_port", "value": "7790"} v 0) v1 -- 0x7f641014bfe0 con 0x7f63f40082c0
2019-01-25T12:11:19.581 INFO:tasks.ceph.mgr.y.mira018.stderr:[25/Jan/2019:12:11:19] ENGINE Bus STARTING 2019-01-25T12:11:19.582 INFO:tasks.ceph.mgr.y.mira018.stderr:CherryPy Checker: 2019-01-25T12:11:19.582 INFO:tasks.ceph.mgr.y.mira018.stderr:The Application mounted at '' has an empty config. 2019-01-25T12:11:19.582 INFO:tasks.ceph.mgr.y.mira018.stderr: 2019-01-25T12:11:19.582 INFO:tasks.ceph.mgr.y.mira018.stderr:[25/Jan/2019:12:11:19] ENGINE Started monitor thread '_TimeoutMonitor'. 2019-01-25T12:11:19.691 INFO:tasks.ceph.mgr.y.mira018.stderr:[25/Jan/2019:12:11:19] ENGINE Serving on :::7790 2019-01-25T12:11:19.722 INFO:tasks.ceph.mgr.y.mira018.stderr:[25/Jan/2019:12:11:19] ENGINE Bus STARTED
2019-01-25T12:11:20.252 INFO:tasks.cephfs_test_runner:====================================================================== 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:ERROR: test_standby (tasks.mgr.test_prometheus.TestPrometheus) 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/mgr/test_prometheus.py", line 48, in test_standby 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner: r = requests.get(original_uri, allow_redirects=False) 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 70, in get 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner: return request('get', url, params=params, **kwargs) 2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 56, in request 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: return session.request(method=method, url=url, **kwargs) 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: resp = self.send(prep, **send_kwargs) 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: r = adapter.send(request, **kwargs) 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 487, in send 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner: raise ConnectionError(e, request=request) 2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner:ConnectionError: HTTPConnectionPool(host='mira018.front.sepia.ceph.com', port=7790): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f22994773d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
This was probably a hard to reproduceable timing error
Related issues
History
#1 Updated by Sebastian Wagner about 5 years ago
Failed again:
DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s INFO:tasks.mgr.mgr_test_case:Found prometheus at http://mira039.front.sepia.ceph.com:7790/ (daemon y/4816) INFO:tasks.mgr.test_prometheus:After failover running at http://mira039.front.sepia.ceph.com:7790/ INFO:tasks.cephfs_test_runner:test_standby (tasks.mgr.test_prometheus.TestPrometheus) ... ERROR INFO:teuthology.orchestra.run.mira053:Running: INFO:teuthology.orchestra.run.mira053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.mgr.test_prometheus.TestPrometheus.test_standby' INFO:[20/Feb/2019:11:14:30] ENGINE Bus STARTING INFO:CherryPy Checker: INFO:The Application mounted at '' has an empty config. INFO: INFO:[20/Feb/2019:11:14:30] ENGINE Started monitor thread '_TimeoutMonitor'. INFO:[20/Feb/2019:11:14:30] ENGINE Serving on :::7791 INFO:[20/Feb/2019:11:14:30] ENGINE Bus STARTED
====================================================================== ERROR: test_standby (tasks.mgr.test_prometheus.TestPrometheus) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_ssh-orch/qa/tasks/mgr/test_prometheus.py", line 48, in test_standby r = requests.get(original_uri, allow_redirects=False) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 70, in get return request('get', url, params=params, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 56, in request return session.request(method=method, url=url, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request resp = self.send(prep, **send_kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send r = adapter.send(request, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 487, in send raise ConnectionError(e, request=request) ConnectionError: HTTPConnectionPool(host='mira039.front.sepia.ceph.com', port=7791): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fdc6623edd0>: Failed to establish a new connection: [Errno 111] Connection refused',))
#2 Updated by Sebastian Wagner about 5 years ago
- Related to Bug #38362: mgr/dashboard: QA run failures: test_standby (tasks.mgr.test_dashboard.TestDashboard) added
#3 Updated by Sebastian Wagner about 5 years ago
- Source changed from Community (dev) to Q/A
- Affected Versions v14.0.0 added
- ceph-qa-suite rados added