Project

General

Profile

Bug #38048

Teuthology error: mgr/prometheus fails with NewConnectionError

Added by Sebastian Wagner 9 months ago. Updated 7 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
prometheus module
Target version:
-
Start date:
01/25/2019
Due date:
% Done:

0%

Source:
Q/A
Tags:
QA
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature:

Description

[[http://qa-proxy.ceph.com/teuthology/swagner-2019-01-24_10:15:07-rados:mgr-wip-swagner-testing-distro-basic-mira/3501097/teuthology.log]]

2019-01-25T12:09:53.069 INFO:teuthology.orchestra.run.mira069.stderr:2019-01-25 12:09:53.064 7f6415f2b700  1 -- v1:172.21.6.110:46408/2431303140 --> [v2:172.21.6.110:3300/0,v1:172.21.6.110:6789/0] -- mon_command({"prefix": "config set", "who": "mgr", "name": "mgr/prometheus/y/server_port", "value": "7790"} v 0) v1 -- 0x7f641014bfe0 con 0x7f63f40082c0
2019-01-25T12:11:19.581 INFO:tasks.ceph.mgr.y.mira018.stderr:[25/Jan/2019:12:11:19] ENGINE Bus STARTING
2019-01-25T12:11:19.582 INFO:tasks.ceph.mgr.y.mira018.stderr:CherryPy Checker:
2019-01-25T12:11:19.582 INFO:tasks.ceph.mgr.y.mira018.stderr:The Application mounted at '' has an empty config.
2019-01-25T12:11:19.582 INFO:tasks.ceph.mgr.y.mira018.stderr:
2019-01-25T12:11:19.582 INFO:tasks.ceph.mgr.y.mira018.stderr:[25/Jan/2019:12:11:19] ENGINE Started monitor thread '_TimeoutMonitor'.
2019-01-25T12:11:19.691 INFO:tasks.ceph.mgr.y.mira018.stderr:[25/Jan/2019:12:11:19] ENGINE Serving on :::7790
2019-01-25T12:11:19.722 INFO:tasks.ceph.mgr.y.mira018.stderr:[25/Jan/2019:12:11:19] ENGINE Bus STARTED
2019-01-25T12:11:20.252 INFO:tasks.cephfs_test_runner:======================================================================
2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:ERROR: test_standby (tasks.mgr.test_prometheus.TestPrometheus)
2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner-testing/qa/tasks/mgr/test_prometheus.py", line 48, in test_standby
2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:    r = requests.get(original_uri, allow_redirects=False)
2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 70, in get
2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:    return request('get', url, params=params, **kwargs)
2019-01-25T12:11:20.253 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner:    return session.request(method=method, url=url, **kwargs)
2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner:    resp = self.send(prep, **send_kwargs)
2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner:    r = adapter.send(request, **kwargs)
2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 487, in send
2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner:    raise ConnectionError(e, request=request)
2019-01-25T12:11:20.254 INFO:tasks.cephfs_test_runner:ConnectionError: HTTPConnectionPool(host='mira018.front.sepia.ceph.com', port=7790): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f22994773d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

This was probably a hard to reproduceable timing error


Related issues

Related to mgr - Bug #38362: mgr/dashboard: QA run failures: test_standby (tasks.mgr.test_dashboard.TestDashboard) New 02/18/2019

History

#1 Updated by Sebastian Wagner 8 months ago

Failed again:

DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s
INFO:tasks.mgr.mgr_test_case:Found prometheus at http://mira039.front.sepia.ceph.com:7790/ (daemon y/4816)
INFO:tasks.mgr.test_prometheus:After failover running at http://mira039.front.sepia.ceph.com:7790/
INFO:tasks.cephfs_test_runner:test_standby (tasks.mgr.test_prometheus.TestPrometheus) ... ERROR
INFO:teuthology.orchestra.run.mira053:Running:
INFO:teuthology.orchestra.run.mira053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.mgr.test_prometheus.TestPrometheus.test_standby'
INFO:[20/Feb/2019:11:14:30] ENGINE Bus STARTING
INFO:CherryPy Checker:
INFO:The Application mounted at '' has an empty config.
INFO:
INFO:[20/Feb/2019:11:14:30] ENGINE Started monitor thread '_TimeoutMonitor'.
INFO:[20/Feb/2019:11:14:30] ENGINE Serving on :::7791
INFO:[20/Feb/2019:11:14:30] ENGINE Bus STARTED

======================================================================
ERROR: test_standby (tasks.mgr.test_prometheus.TestPrometheus)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_ceph-c_ssh-orch/qa/tasks/mgr/test_prometheus.py", line 48, in test_standby
    r = requests.get(original_uri, allow_redirects=False)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 487, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='mira039.front.sepia.ceph.com', port=7791): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fdc6623edd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

#2 Updated by Sebastian Wagner 8 months ago

  • Related to Bug #38362: mgr/dashboard: QA run failures: test_standby (tasks.mgr.test_dashboard.TestDashboard) added

#3 Updated by Sebastian Wagner 8 months ago

  • Source changed from Community (dev) to Q/A
  • Affected Versions v14.0.0 added
  • ceph-qa-suite rados added

Also available in: Atom PDF