Ceph » mgr » Dashboard

I've seen the following error a few times in different QA runs:

2019-02-14T11:32:54.435 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.351 7f24477fe700 20 mgrc handle_mgr_map mgrmap(e 21) v1
2019-02-14T11:32:54.435 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.351 7f24477fe700  4 mgrc handle_mgr_map Got map version 21
2019-02-14T11:32:54.435 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.351 7f24477fe700  4 mgrc handle_mgr_map Active mgr is now [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318]
2019-02-14T11:32:54.435 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.351 7f24477fe700  4 mgrc reconnect Starting new session with [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318]
2019-02-14T11:32:54.436 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.351 7f24477fe700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318] conn(0x7f243401d0b0 0x7f243401f560 :-1 s=NONE pgs=0 cs=0 l=1).connect
2019-02-14T11:32:54.437 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.351 7f24477fe700  1 -- 172.21.6.138:0/1035006712 <== mon.1 v2:172.21.7.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5498+0+0 (943254598 0 0) 0x7f243803a1a0 con 0x7f24500d8de0
2019-02-14T11:32:54.437 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.352 7f244e621700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318] conn(0x7f243401d0b0 0x7f243401f560 :-1 s=CONNECTING pgs=0 cs=0 l=1)._handle_peer_banner_payload supported=0 required=0
2019-02-14T11:32:54.437 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.353 7f244e621700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318] conn(0x7f243401d0b0 0x7f243401f560 :-1 s=READY pgs=17 cs=0 l=1).ready entity=mgr.4750 cookie=0 in_seq=0 out_seq=0
2019-02-14T11:32:54.437 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.384 7f2455081700  1 -- 172.21.6.138:0/1035006712 --> [v2:172.21.7.118:3300/0,v1:172.21.7.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2450064d00 con 0x7f24500d8de0
2019-02-14T11:32:54.437 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.399 7f24477fe700  1 -- 172.21.6.138:0/1035006712 <== mon.1 v2:172.21.7.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0  v0) v1 ==== 72+0+93695 (1092875540 0 2918681843) 0x7f2438004e90 con 0x7f24500d8de0
2019-02-14T11:32:54.937 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:54.930 7f2455081700  1 -- 172.21.6.138:0/1035006712 --> [v2:172.21.7.118:3300/0,v1:172.21.7.118:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.mgr.test_dashboard.TestDashboard.test_standby"]} v 0) v1 -- 0x7f2450064d00 con 0x7f24500d8de0
2019-02-14T11:32:55.064 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.057 7f24477fe700  1 -- 172.21.6.138:0/1035006712 <== mon.1 v2:172.21.7.118:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.mgr.test_dashboard.TestDashboard.test_standby"]}]=0  v159) v1 ==== 130+0+0 (2881010698 0 0) 0x7f24380050a0 con 0x7f24500d8de0
2019-02-14T11:32:55.106 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700 10 mgrc shutdown
2019-02-14T11:32:55.107 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 -- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318] conn(0x7f243401d0b0 msgr2=0x7f243401f560 :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2019-02-14T11:32:55.107 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318] conn(0x7f243401d0b0 0x7f243401f560 :-1 s=READY pgs=17 cs=0 l=1).stop
2019-02-14T11:32:55.107 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 -- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:3300/0,v1:172.21.7.118:6789/0] conn(0x7f24500d8de0 msgr2=0x7f24500d9240 :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2019-02-14T11:32:55.107 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:3300/0,v1:172.21.7.118:6789/0] conn(0x7f24500d8de0 0x7f24500d9240 :-1 s=READY pgs=213 cs=0 l=1).stop
2019-02-14T11:32:55.108 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 -- 172.21.6.138:0/1035006712 shutdown_connections
2019-02-14T11:32:55.108 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:6828/34318,v1:172.21.7.118:6829/34318] conn(0x7f243401d0b0 0x7f243401f560 :-1 s=CLOSED pgs=17 cs=0 l=1).stop
2019-02-14T11:32:55.108 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.100 7f2455081700  1 --2- 172.21.6.138:0/1035006712 >> [v2:172.21.7.118:3300/0,v1:172.21.7.118:6789/0] conn(0x7f24500d8de0 0x7f24500d9240 :-1 s=CLOSED pgs=213 cs=0 l=1).stop
2019-02-14T11:32:55.108 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.101 7f2455081700  1 -- 172.21.6.138:0/1035006712 shutdown_connections
2019-02-14T11:32:55.108 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.101 7f2455081700  1 -- 172.21.6.138:0/1035006712 wait complete.
2019-02-14T11:32:55.108 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:32:55.101 7f2455081700  1 -- 172.21.6.138:0/1035006712 >> 172.21.6.138:0/1035006712 conn(0x7f24500b1640 msgr2=0x7f2450093bf0 :-1 s=STATE_NONE l=0).mark_down
2019-02-14T11:32:55.124 INFO:tasks.cephfs_test_runner:
2019-02-14T11:32:55.124 INFO:tasks.cephfs_test_runner:======================================================================
2019-02-14T11:32:55.124 INFO:tasks.cephfs_test_runner:ERROR: test_standby (tasks.mgr.test_dashboard.TestDashboard)
2019-02-14T11:32:55.124 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-26378/qa/tasks/mgr/test_dashboard.py", line 38, in test_standby
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:    r = requests.get(original_uri, allow_redirects=False, verify=False)
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 70, in get
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:    return request('get', url, params=params, **kwargs)
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:    return session.request(method=method, url=url, **kwargs)
2019-02-14T11:32:55.125 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:    resp = self.send(prep, **send_kwargs)
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:    r = adapter.send(request, **kwargs)
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 487, in send
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:    raise ConnectionError(e, request=request)
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:ConnectionError: HTTPSConnectionPool(host='mira083.front.sepia.ceph.com', port=7791): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f70a36fdb50>: Failed to establish a new connection: [Errno 111] Connection refused',))
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-02-14T11:32:55.126 INFO:tasks.cephfs_test_runner:Ran 1 test in 73.118s
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:======================================================================
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:ERROR: test_standby (tasks.mgr.test_dashboard.TestDashboard)
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-02-14T11:32:55.127 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-26378/qa/tasks/mgr/test_dashboard.py", line 38, in test_standby
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:    r = requests.get(original_uri, allow_redirects=False, verify=False)
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 70, in get
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:    return request('get', url, params=params, **kwargs)
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:    return session.request(method=method, url=url, **kwargs)
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:    resp = self.send(prep, **send_kwargs)
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
2019-02-14T11:32:55.128 INFO:tasks.cephfs_test_runner:    r = adapter.send(request, **kwargs)
2019-02-14T11:32:55.129 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 487, in send
2019-02-14T11:32:55.129 INFO:tasks.cephfs_test_runner:    raise ConnectionError(e, request=request)
2019-02-14T11:32:55.129 INFO:tasks.cephfs_test_runner:ConnectionError: HTTPSConnectionPool(host='mira083.front.sepia.ceph.com', port=7791): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f70a36fdb50>: Failed to establish a new connection: [Errno 111] Connection refused',))
2019-02-14T11:32:55.129 INFO:tasks.cephfs_test_runner:
2019-02-14T11:32:55.131 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-26378/qa/tasks/cephfs_test_runner.py", line 207, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_standby (tasks.mgr.test_dashboard.TestDashboard)
2019-02-14T11:32:55.131 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner
2019-02-14T11:32:55.146 DEBUG:teuthology.run_tasks:Unwinding manager rgw
2019-02-14T11:32:55.160 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 32, in nested
    yield vars
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-26378/qa/tasks/rgw.py", line 347, in task
    yield
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-26378/qa/tasks/cephfs_test_runner.py", line 207, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_standby (tasks.mgr.test_dashboard.TestDashboard)
2019-02-14T11:32:55.161 DEBUG:tasks.rgw.client.0:waiting for process to exit
2019-02-14T11:32:55.161 INFO:teuthology.orchestra.run:waiting for 300
2019-02-14T11:32:55.168 INFO:tasks.rgw.client.0.mira056.stdout:2019-02-14 11:32:55.158 7f503f7ea700 -1 received  signal: Terminated from /usr/bin/python /bin/daemon-helper term radosgw --rgw-frontends civetweb port=7280 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground  (PID: 35633) UID: 0
2019-02-14T11:32:55.169 INFO:tasks.rgw.client.0.mira056.stdout:2019-02-14 11:32:55.158 7f5054994f00 -1 shutting down
2019-02-14T11:33:01.262 INFO:tasks.rgw.client.0:Stopped
2019-02-14T11:33:01.263 INFO:teuthology.orchestra.run.mira056:Running:
2019-02-14T11:33:01.263 INFO:teuthology.orchestra.run.mira056:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock
2019-02-14T11:33:01.380 DEBUG:teuthology.run_tasks:Unwinding manager ceph
2019-02-14T11:33:01.650 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges...
2019-02-14T11:33:01.650 INFO:teuthology.orchestra.run.mira056:Running:
2019-02-14T11:33:01.650 INFO:teuthology.orchestra.run.mira056:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json
2019-02-14T11:33:02.051 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:33:02.043 7f5106b91700  1  Processor -- start
2019-02-14T11:33:02.051 INFO:teuthology.orchestra.run.mira056.stderr:2019-02-14 11:33:02.044 7f5106b91700  1 --  start start

also see: http://qa-proxy.ceph.com/teuthology/laura-2019-02-14_10:59:29-rados:mgr-wip-lpaduano-testing-26378-distro-basic-mira/3590821/teuthology.log