Bug #56072
Updated by Sridhar Seshasayee almost 2 years ago
h3. Description of problem
Hit this in the following teuthology run on pacific:
/a/yuriw-2022-06-15_18:29:33-rados-wip-yuri4-testing-2022-06-15-1000-pacific-distro-default-smithi/6881164
Test Description:
rados/dashboard/{centos_8.stream_container_tools clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_8} tasks/dashboard}
h3. Environment
* @ceph version@ string: pacific
* Platform (OS/distro/release):
* Cluster details (nodes, monitors, OSDs):
* Did it happen on a stable environment or after a migration/upgrade?:
Observed on pacific teuthology test run. Test description:
rados/dashboard/{centos_8.stream_container_tools clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_8} tasks/dashboard}
* Browser used (e.g.: @Version 86.0.4240.198 (Official Build) (64-bit)@):
h3. Actual results
<pre>
2022-06-15T21:29:59.531 DEBUG:tasks.mgr.dashboard.helper:Request PUT to https://172.21.15.87:7789/api/telemetry
2022-06-15T21:29:59.559 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.87:7789/api/telemetry/report
2022-06-15T21:29:59.579 ERROR:tasks.mgr.dashboard.helper:Request response: {"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "83a9fb2e-6259-4ae0-a70e-d55c30e80726"}
2022-06-15T21:29:59.581 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.mgr.dashboard.test_telemetry.TelemetryTest.test_get_report'
2022-06-15T21:29:59.583 INFO:tasks.ceph.mgr.x.smithi087.stderr:2022-06-15T21:29:59.572+0000 7fdbed88c700 -1 mgr get_metadata_python Requested missing service mon.b
2022-06-15T21:29:59.584 INFO:tasks.ceph.mgr.x.smithi087.stderr:2022-06-15T21:29:59.572+0000 7fdbed88c700 -1 Remote method threw exception: Traceback (most recent call last):
2022-06-15T21:29:59.584 INFO:tasks.ceph.mgr.x.smithi087.stderr: File "/usr/share/ceph/mgr/telemetry/module.py", line 800, in get_report
2022-06-15T21:29:59.584 INFO:tasks.ceph.mgr.x.smithi087.stderr: return {'report': self.compile_report(channels=channels),
2022-06-15T21:29:59.586 INFO:tasks.ceph.mgr.x.smithi087.stderr: File "/usr/share/ceph/mgr/telemetry/module.py", line 623, in compile_report
2022-06-15T21:29:59.587 INFO:tasks.ceph.mgr.x.smithi087.stderr: report['metadata']['mon'] = self.gather_mon_metadata(mon_map)
2022-06-15T21:29:59.587 INFO:tasks.ceph.mgr.x.smithi087.stderr: File "/usr/share/ceph/mgr/telemetry/module.py", line 266, in gather_mon_metadata
2022-06-15T21:29:59.587 INFO:tasks.ceph.mgr.x.smithi087.stderr: res = self.get_metadata('mon', mon['name']).items()
2022-06-15T21:29:59.588 INFO:tasks.ceph.mgr.x.smithi087.stderr:AttributeError: 'NoneType' object has no attribute 'items'
...
...
2022-06-15T21:30:00.194 INFO:tasks.cephfs_test_runner:test_get_report (tasks.mgr.dashboard.test_telemetry.TelemetryTest) ... FAIL
2022-06-15T21:30:00.195 DEBUG:tasks.mgr.dashboard.helper:Request PUT to https://172.21.15.87:7789/api/telemetry
2022-06-15T21:30:00.220 INFO:tasks.cephfs_test_runner:
2022-06-15T21:30:00.221 INFO:tasks.cephfs_test_runner:======================================================================
2022-06-15T21:30:00.221 INFO:tasks.cephfs_test_runner:FAIL: test_get_report (tasks.mgr.dashboard.test_telemetry.TelemetryTest)
2022-06-15T21:30:00.221 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-06-15T21:30:00.222 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-06-15T21:30:00.222 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_224fc22e07cebeecc3e08055cfd6105b1a30f173/qa/tasks/mgr/dashboard/test_telemetry.py", line 75, in test_get_report
2022-06-15T21:30:00.222 INFO:tasks.cephfs_test_runner: self.assertStatus(200)
2022-06-15T21:30:00.223 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_224fc22e07cebeecc3e08055cfd6105b1a30f173/qa/tasks/mgr/dashboard/helper.py", line 495, in assertStatus
2022-06-15T21:30:00.223 INFO:tasks.cephfs_test_runner: self.assertEqual(self._resp.status_code, status)
2022-06-15T21:30:00.223 INFO:tasks.cephfs_test_runner:AssertionError: 500 != 200
2022-06-15T21:30:00.223 INFO:tasks.cephfs_test_runner:
2022-06-15T21:30:00.224 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-06-15T21:30:00.224 INFO:tasks.cephfs_test_runner:Ran 245 tests in 6097.959s
2022-06-15T21:30:00.224 INFO:tasks.cephfs_test_runner:
2022-06-15T21:30:00.225 INFO:tasks.cephfs_test_runner:FAILED (failures=1)
2022-06-15T21:30:00.225 INFO:tasks.cephfs_test_runner:
2022-06-15T21:30:00.225 INFO:tasks.cephfs_test_runner:=====================================================================
</pre>