Ceph » mgr » Dashboard

Description of problem¶

```
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: Internal Server Error
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: Traceback (most recent call last):
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: return handler(*args, **kwargs)
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in call
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: return self.callable(*self.args, **self.kwargs)
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 263, in inner
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: ret = func(*args, **kwargs)
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 191, in wrapper
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: return func(*vpath, **params)
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: File "/usr/share/ceph/mgr/dashboard/controllers/osd.py", line 173, in settings
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: 'nearfull_ratio': result['nearfull_ratio'],
Jul 17 01:54:27 ceph-node-01 ceph-fd8f5030-2424-11ee-9146-525400983882-mgr-ceph-node-01-fmzyoi<sup>2273</sup>: TypeError: string indices must be integers
```

I see it when OSDs are being created, which in turn leads to a remap and backfil

Given the error it appears that the response is not a dict but a byte/string. I added a log call in the send_command function of ceph_service.py to check and the try except clause when converting to json is hitting the except and returning a str. See 2nd line below.

Jul 17 20:14:04 ceph-node-02 ceph-mgr²¹²⁷: [dashboard INFO request] [::ffff:172.18.101.1:50930] [GET] [200] [0.072s] [admin] [97.0B] /api/health/get_cluster_capacity
Jul 17 20:14:04 ceph-node-02 ceph-mgr²¹²⁷: [dashboard ERROR ceph_service] unable to load command response as json
Jul 17 20:14:04 ceph-node-02 ceph-mgr²¹²⁷: [dashboard ERROR exception] Internal Server Error
Traceback (most recent call last):
File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
return handler(args, **kwargs)
File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in call
return self.callable(*self.args, **self.kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 263, in inner
ret = func(*args, **kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 191, in wrapper
return func(*vpath, **params)
File "/usr/share/ceph/mgr/dashboard/controllers/osd.py", line 173, in settings
'nearfull_ratio': result['nearfull_ratio'],
TypeError: string indices must be integers
Jul 17 20:14:04 ceph-node-02 ceph-mgr²¹²⁷: [dashboard ERROR request] [::ffff:172.18.101.1:50922] [GET] [500] [1.702s] [admin] [513.0B] /api/osd/settings
Jul 17 20:14:04 ceph-node-02 ceph-mgr²¹²⁷: [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "
b3fb6df8-4bfa-4774-b173-1e0552a3a1d3"}
']
It looks to my like the stats for .mgr pool are giving an invalid value 'inf' for pools[].application_metadata.read_balance.score_acting and score_stable - which is why the decode fails and the settings call fails.

,"application_metadata":{"mgr":{}},"read_balance":{"score_acting":inf,"score_stable":inf,

 I just checked on 18.1.2-68 and this issue is not present - so it's probably related to when the branch was made (although the system I compared with only had 3 osds, not 750! - and this issue wasn't present at the start of the test...) 

Environment¶

• ceph version string:
• Platform (OS/distro/release):
• Cluster details (nodes, monitors, OSDs):
• Did it happen on a stable environment or after a migration/upgrade?:
• Browser used (e.g.: Version 86.0.4240.198 (Official Build) (64-bit)):

How reproducible¶

Steps:

3. ...

Actual results¶

Please add logs and/or screenshots

Expected results¶

here

Additional info¶

here