Ceph » CephFS

tasks.cephfs.test_admin.TestAdminCommandDumpLoads.test_dump_loads fails with JSONDecodeError. Traceback -

======================================================================
ERROR: test_dump_loads (tasks.cephfs.test_admin.TestAdminCommandDumpLoads)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/cephfs/test_admin.py", line 1408, in test_dump_loads
    loads = self.fs.mds_asok(['dump', 'loads', '1'])
  File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/cephfs/filesystem.py", line 1179, in mds_asok
    return self.rank_asok(command, timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/cephfs/filesystem.py", line 1191, in rank_asok
    return self.json_asok(command, 'mds', info['name'], timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/cephfs/filesystem.py", line 258, in json_asok
    j = json.loads(response_data)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads 
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode 
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 940 (char 939)

Introduced by: https://github.com/ceph/ceph/pull/46685/commits/a8f138cd3623e42c4f74a18d54ec7dfcc1e89e58

The error was seen in following QA runs -
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910713/
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-03_15:10:28-fs-wip-rishabh-testing-20220702-1631-distro-default-smithi/6911952/

The issue can reproduced with vstart_runner.py as well.

'{"dirfrags":[{"path":"","dirfrag":"0x1","pop_me":{"Decay Counters":[{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0.23618757717452724,"halflife":5}]},"pop_nested":{"Decay Counters":[{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0.23618757717452724,"halflife":5}]},"pop_auth_subtree":{"Decay Counters":[{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0.23618757717452724,"halflife":5}]},"pop_auth_subtree_nested":{"Decay Counters":[{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0.23618757717452724,"halflife":5}]}}],"mds_load":{"mds.0":{"request_rate":0,"cache_hit_rate":0,"queue_length":0,"cpu_load":0,"mds_load":1.9348242315747266,"auth_dirfrags":{"Decay Counters":[{"value":0,"halflife":-inf},{"value":0,"halflife":-inf},{"value":0,"halflife":-inf},{"value":0,"halflife":-inf},{"value":0.48370605789368165,"halflife":-inf}]},"all_dirfrags":{"Decay Counters":[{"value":0,"halflife":-inf},{"value":0,"halflife":-inf},{"value":0,"halflife":-inf},{"value":0,"halflife":-inf},{"value":0.48370605789368165,"halflife":-inf}]}}},"mds_meta_load":{},"mds_import_map":{"mds.0":[]}}'