Bug #38362: mgr/dashboard: QA run failures: test_standby (tasks.mgr.test_dashboard.TestDashboard) - Dashboard - Ceph

Actions

Copy link

Bug #38362

closed

mgr/dashboard: QA run failures: test_standby (tasks.mgr.test_dashboard.TestDashboard)

Added by Laura Paduano about 5 years ago. Updated about 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Category:

Testing & QA

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v14.0.0

ceph-qa-suite:

rados

Pull request ID:

26986

Crash signature (v1):

Crash signature (v2):

Description

I've seen the following error a few times in different QA runs:

2019-02-14T11:32:54.435 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.351 7f24477fe700 20 mgrc handle_mgr_map mgrmap(e 21) v1
2019-02-14T11:32:54.435 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.351 7f24477fe700  4 mgrc handle_mgr_map Got map version 21
2019-02-14T11:32:54.435 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.351 7f24477fe700  4 mgrc handle_mgr_map Active mgr is now [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318]
2019-02-14T11:32:54.435 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.351 7f24477fe700  4 mgrc reconnect Starting new session with [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318]
2019-02-14T11:32:54.436 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.351 7f24477fe700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318] conn(0x7f243401d0b0 0x7f243401f560 :-1 s=NONE pgs=0 cs=0 l=1).connect
2019-02-14T11:32:54.437 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.351 7f24477fe700  1 -- 172.21.6.138:0/1035006712 <== mon.1 v2:172.21.7.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5498+0+0 (943254598 0 0) 0x7f243803a1a0 con 0x7f24500d8de0
2019-02-14T11:32:54.437 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.352 7f244e621700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318] conn(0x7f243401d0b0 0x7f243401f560 :-1 s=CONNECTING pgs=0 cs=0 l=1)._handle_peer_banner_payload supported=0 required=0
2019-02-14T11:32:54.437 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.353 7f244e621700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318] conn(0x7f243401d0b0 0x7f243401f560 :-1 s=READY pgs=17 cs=0 l=1).ready entity=mgr.4750 cookie=0 in_seq=0 out_seq=0
2019-02-14T11:32:54.437 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.384 7f2455081700  1 -- 172.21.6.138:0/1035006712 --> [v2:172.21.7.118:3300/0,v1:172.21.7.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2450064d00 con 0x7f24500d8de0
2019-02-14T11:32:54.437 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.399 7f24477fe700  1 -- 172.21.6.138:0/1035006712 <== mon.1 v2:172.21.7.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0  v0) v1 ==== 72+0+93695 (1092875540 0 2918681843) 0x7f2438004e90 con 0x7f24500d8de0
2019-02-14T11:32:54.937 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.930 7f2455081700  1 -- 172.21.6.138:0/1035006712 --> [v2:172.21.7.118:3300/0,v1:172.21.7.118:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.mgr.test_dashboard.TestDashboard.test_standby"]} v 0) v1 -- 0x7f2450064d00 con 0x7f24500d8de0
2019-02-14T11:32:55.064 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.057 7f24477fe700  1 -- 172.21.6.138:0/1035006712 <== mon.1 v2:172.21.7.118:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.mgr.test_dashboard.TestDashboard.test_standby"]}]=0  v159) v1 ==== 130+0+0 (2881010698 0 0) 0x7f24380050a0 con 0x7f24500d8de0
2019-02-14T11:32:55.106 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700 10 mgrc shutdown
2019-02-14T11:32:55.107 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 -- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318] conn(0x7f243401d0b0 msgr2=0x7f243401f560 :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2019-02-14T11:32:55.107 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318] conn(0x7f243401d0b0 0x7f243401f560 :-1 s=READY pgs=17 cs=0 l=1).stop
2019-02-14T11:32:55.107 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 -- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:3300/0,v1:172.21.7.118:6789/0] conn(0x7f24500d8de0 msgr2=0x7f24500d9240 :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2019-02-14T11:32:55.107 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:3300/0,v1:172.21.7.118:6789/0] conn(0x7f24500d8de0 0x7f24500d9240 :-1 s=READY pgs=213 cs=0 l=1).stop
2019-02-14T11:32:55.108 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 -- 172.21.6.138:0/1035006712 shutdown_connections
2019-02-14T11:32:55.108 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318] conn(0x7f243401d0b0 0x7f243401f560 :-1 s=CLOSED pgs=17 cs=0 l=1).stop
2019-02-14T11:32:55.108 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:3300/0,v1:172.21.7.118:6789/0] conn(0x7f24500d8de0 0x7f24500d9240 :-1 s=CLOSED pgs=213 cs=0 l=1).stop
2019-02-14T11:32:55.108 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.101 7f2455081700  1 -- 172.21.6.138:0/1035006712 shutdown_connections
2019-02-14T11:32:55.108 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.101 7f2455081700  1 -- 172.21.6.138:0/1035006712 wait complete.
2019-02-14T11:32:55.108 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.101 7f2455081700  1 -- 172.21.6.138:0/1035006712 >> 172.21.6.138:0/1035006712 conn(0x7f24500b1640 msgr2=0x7f2450093bf0 :-1 s=STATE_NONE l=0).mark_down
2019-02-14T11:32:55.124 INFO:tasks.cephfs_test_runner:
2019-02-14T11:32:55.124 INFO:tasks.cephfs_test_runner:======================================================================
2019-02-14T11:32:55.124 INFO:tasks.cephfs_test_runner:ERROR: test_standby (tasks.mgr.test_dashboard.TestDashboard)
2019-02-14T11:32:55.124 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-26378/qa/tasks/mgr/test_dashboard.py", line 38, in test_standby
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:    r = requests.get(original_uri, allow_redirects=False, verify=False)
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 70, in get
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:    return request('get', url, params=params, **kwargs)
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:    return session.request(method=method, url=url, **kwargs)
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:    resp = self.send(prep, **send_kwargs)
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:    r = adapter.send(request, **kwargs)
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 487, in send
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:    raise ConnectionError(e, request=request)
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:ConnectionError: HTTPSConnectionPool(host='mira083.front.sepia.ceph.com', port=7791): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f70a36fdb50>: Failed to establish a new connection: [Errno 111] Connection refused',))
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:Ran 1 test in 73.118s
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:======================================================================
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:ERROR: test_standby (tasks.mgr.test_dashboard.TestDashboard)
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-26378/qa/tasks/mgr/test_dashboard.py", line 38, in test_standby
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:    r = requests.get(original_uri, allow_redirects=False, verify=False)
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 70, in get
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:    return request('get', url, params=params, **kwargs)
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:    return session.request(method=method, url=url, **kwargs)
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:    resp = self.send(prep, **send_kwargs)
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:    r = adapter.send(request, **kwargs)
2019-02-14T11:32:55.129 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 487, in send
2019-02-14T11:32:55.129 INFO:tasks.cephfs_test_runner:    raise ConnectionError(e, request=request)
2019-02-14T11:32:55.129 INFO:tasks.cephfs_test_runner:ConnectionError: HTTPSConnectionPool(host='mira083.front.sepia.ceph.com', port=7791): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f70a36fdb50>: Failed to establish a new connection: [Errno 111] Connection refused',))
2019-02-14T11:32:55.129 INFO:tasks.cephfs_test_runner:
2019-02-14T11:32:55.131 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-26378/qa/tasks/cephfs_test_runner.py", line 207, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_standby (tasks.mgr.test_dashboard.TestDashboard)
2019-02-14T11:32:55.131 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner
2019-02-14T11:32:55.146 DEBUG:teuthology.run_tasks:Unwinding manager rgw
2019-02-14T11:32:55.160 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 32, in nested
    yield vars
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-26378/qa/tasks/rgw.py", line 347, in task
    yield
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-26378/qa/tasks/cephfs_test_runner.py", line 207, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_standby (tasks.mgr.test_dashboard.TestDashboard)
2019-02-14T11:32:55.161 DEBUG:tasks.rgw.client.0:waiting for process to exit
2019-02-14T11:32:55.161 INFO:teuthology.orchestra.run:waiting for 300
2019-02-14T11:32:55.168 INFO:tasks.rgw.client.0.mira056.stdout:2019-02-14 11:32:55.158 7f503f7ea700 -1 received  signal: Terminated from /usr/bin/python /bin/daemon-helper term radosgw --rgw-frontends civetweb port=7280 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground  (PID: 35633) UID: 0
2019-02-14T11:32:55.169 INFO:tasks.rgw.client.0.mira056.stdout:2019-02-14 11:32:55.158 7f5054994f00 -1 shutting down
2019-02-14T11:33:01.262 INFO:tasks.rgw.client.0:Stopped
2019-02-14T11:33:01.263 INFO:teuthology.orchestra.run.mira056:Running:
2019-02-14T11:33:01.263 INFO:teuthology.orchestra.run.mira056:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock
2019-02-14T11:33:01.380 DEBUG:teuthology.run_tasks:Unwinding manager ceph
2019-02-14T11:33:01.650 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges...
2019-02-14T11:33:01.650 INFO:teuthology.orchestra.run.mira056:Running:
2019-02-14T11:33:01.650 INFO:teuthology.orchestra.run.mira056:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json
2019-02-14T11:33:02.051 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:33:02.043 7f5106b91700  1  Processor -- start
2019-02-14T11:33:02.051 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:33:02.044 7f5106b91700  1 --  start start

also see: http://qa-proxy.ceph.com/teuthology/laura-2019-02-14_10:59:29-rados:mgr-wip-lpaduano-testing-26378-distro-basic-mira/3590821/teuthology.log

Related issues 1 (1 open — 0 closed)

Actions

Copy link

Updated by Tatjana Dehler about 5 years ago

Also http://pulpito.ceph.com/tdehler-2019-02-18_09:54:35-rados:mgr-wip-tdehler-testing-26450-distro-basic-mira/

Actions

Copy link

Updated by Sebastian Wagner about 5 years ago

Related to Bug #38048: Teuthology error: mgr/prometheus fails with NewConnectionError added

Actions

Copy link

Updated by Sebastian Wagner about 5 years ago

Both, dashboard and prometheus use the same CherryPy server. If we fix this issue for the dashbaord, the code needs to be to also fix other CherryPy based modules.

Actions

Copy link

Updated by Tatjana Dehler about 5 years ago

I also experienced the issue while running the tests on a vstart cluster (./run-backend-api-tests.sh tasks.mgr.test_dashboard.TestDashboard).

Actions

Copy link

Updated by Volker Theile about 5 years ago

I assume this you'll find

Module 'dashboard' has failed: ceph_get_module_option() takes exactly 2 arguments (1 given)

in the log file. This issue has been fixed with https://github.com/ceph/ceph/pull/26986.

Actions

Copy link

Updated by Laura Paduano over 4 years ago

Status changed from New to Resolved
Pull request ID set to 26986

Actions

Copy link

Updated by Ernesto Puerta about 3 years ago

Project changed from mgr to Dashboard
Category changed from 151 to Testing & QA

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph » mgr » Dashboard

Custom queries

Bug #38362

mgr/dashboard: QA run failures: test_standby (tasks.mgr.test_dashboard.TestDashboard)

Updated by Tatjana Dehler about 5 years ago

Updated by Sebastian Wagner about 5 years ago

Updated by Sebastian Wagner about 5 years ago

Updated by Tatjana Dehler about 5 years ago

Updated by Volker Theile about 5 years ago

Updated by Laura Paduano over 4 years ago

Updated by Ernesto Puerta about 3 years ago