Bug #62089
closedmgr/dashboard: TypeError: string indices must be integers
100%
Description
Description of problem¶
```
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: Internal Server Error
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: Traceback (most recent call last):
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: return handler(*args, **kwargs)
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in call
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: return self.callable(*self.args, **self.kwargs)
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 263, in inner
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: ret = func(*args, **kwargs)
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 191, in wrapper
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: return func(*vpath, **params)
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: File "/usr/share/ceph/mgr/dashboard/controllers/osd.py", line 173, in settings
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: 'nearfull_ratio': result['nearfull_ratio'],
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi2273: TypeError: string indices must be integers
```
I see it when OSDs are being created, which in turn leads to a remap and backfil
Given the error it appears that the response is not a dict but a byte/string. I added a log call in the send_command function of ceph_service.py to check and the try except clause when converting to json is hitting the except and returning a str. See 2nd line below.
Jul 17 20:14:04 ceph-node-02 ceph-mgr2127: [dashboard INFO request] [::ffff:172.18.101.1:50930] [GET] [200] [0.072s] [admin] [97.0B] /api/health/get_cluster_capacity
Jul 17 20:14:04 ceph-node-02 ceph-mgr2127: [dashboard ERROR ceph_service] unable to load command response as json
Jul 17 20:14:04 ceph-node-02 ceph-mgr2127: [dashboard ERROR exception] Internal Server Error
Traceback (most recent call last):
File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
return handler(args, **kwargs)
File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in call
return self.callable(*self.args, **self.kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 263, in inner
ret = func(*args, **kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 191, in wrapper
return func(*vpath, **params)
File "/usr/share/ceph/mgr/dashboard/controllers/osd.py", line 173, in settings
'nearfull_ratio': result['nearfull_ratio'],
TypeError: string indices must be integers
Jul 17 20:14:04 ceph-node-02 ceph-mgr2127: [dashboard ERROR request] [::ffff:172.18.101.1:50922] [GET] [500] [1.702s] [admin] [513.0B] /api/osd/settings
Jul 17 20:14:04 ceph-node-02 ceph-mgr2127: [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "
b3fb6df8-4bfa-4774-b173-1e0552a3a1d3"}
']
It looks to my like the stats for .mgr pool are giving an invalid value 'inf' for pools[].application_metadata.read_balance.score_acting and score_stable - which is why the decode fails and the settings call fails.
,"application_metadata":{"mgr":{}},"read_balance":{"score_acting":inf,"score_stable":inf,
I just checked on 18.1.2-68 and this issue is not present - so it's probably related to when the branch was made (although the system I compared with only had 3 osds, not 750! - and this issue wasn't present at the start of the test...)
Environment¶
ceph version
string:- Platform (OS/distro/release):
- Cluster details (nodes, monitors, OSDs):
- Did it happen on a stable environment or after a migration/upgrade?:
- Browser used (e.g.:
Version 86.0.4240.198 (Official Build) (64-bit)
):
How reproducible¶
Steps:
- ...
Actual results¶
Please add logs and/or screenshots
Expected results¶
here
Additional info¶
here