Project

General

Profile

Bug #57299

qa: test_dump_loads fails with JSONDecodeError

Added by Rishabh Dave 5 months ago. Updated 3 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
Administration/Usability
Target version:
% Done:

0%

Source:
Tags:
backport_processed
Backport:
quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
qa-suite
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

tasks.cephfs.test_admin.TestAdminCommandDumpLoads.test_dump_loads fails with JSONDecodeError. Traceback -

======================================================================
ERROR: test_dump_loads (tasks.cephfs.test_admin.TestAdminCommandDumpLoads)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/cephfs/test_admin.py", line 1408, in test_dump_loads
    loads = self.fs.mds_asok(['dump', 'loads', '1'])
  File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/cephfs/filesystem.py", line 1179, in mds_asok
    return self.rank_asok(command, timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/cephfs/filesystem.py", line 1191, in rank_asok
    return self.json_asok(command, 'mds', info['name'], timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/cephfs/filesystem.py", line 258, in json_asok
    j = json.loads(response_data)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads 
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode 
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 940 (char 939)

Introduced by: https://github.com/ceph/ceph/pull/46685/commits/a8f138cd3623e42c4f74a18d54ec7dfcc1e89e58

The error was seen in following QA runs -
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910713/
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-03_15:10:28-fs-wip-rishabh-testing-20220702-1631-distro-default-smithi/6911952/

The issue can reproduced with vstart_runner.py as well.

'{"dirfrags":[{"path":"","dirfrag":"0x1","pop_me":{"Decay Counters":[{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0.23618757717452724,"halflife":5}]},"pop_nested":{"Decay Counters":[{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0.23618757717452724,"halflife":5}]},"pop_auth_subtree":{"Decay Counters":[{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0.23618757717452724,"halflife":5}]},"pop_auth_subtree_nested":{"Decay Counters":[{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0,"halflife":5},{"value":0.23618757717452724,"halflife":5}]}}],"mds_load":{"mds.0":{"request_rate":0,"cache_hit_rate":0,"queue_length":0,"cpu_load":0,"mds_load":1.9348242315747266,"auth_dirfrags":{"Decay Counters":[{"value":0,"halflife":-inf},{"value":0,"halflife":-inf},{"value":0,"halflife":-inf},{"value":0,"halflife":-inf},{"value":0.48370605789368165,"halflife":-inf}]},"all_dirfrags":{"Decay Counters":[{"value":0,"halflife":-inf},{"value":0,"halflife":-inf},{"value":0,"halflife":-inf},{"value":0,"halflife":-inf},{"value":0.48370605789368165,"halflife":-inf}]}}},"mds_meta_load":{},"mds_import_map":{"mds.0":[]}}'

Related issues

Copied to CephFS - Backport #57929: quincy: qa: test_dump_loads fails with JSONDecodeError New

History

#1 Updated by Rishabh Dave 5 months ago

  • Description updated (diff)
  • Pull request ID set to 47812

#2 Updated by Rishabh Dave 5 months ago

  • Status changed from In Progress to Fix Under Review

#3 Updated by Rishabh Dave 5 months ago

  • Description updated (diff)

#4 Updated by Rishabh Dave 5 months ago

  • Description updated (diff)

#5 Updated by Venky Shankar 4 months ago

Rishabh, should this change be backported to p/q releases?

#6 Updated by Venky Shankar 3 months ago

  • Category set to Administration/Usability
  • Status changed from Fix Under Review to Pending Backport
  • Target version set to v18.0.0
  • Backport set to quincy

#7 Updated by Backport Bot 3 months ago

  • Copied to Backport #57929: quincy: qa: test_dump_loads fails with JSONDecodeError added

#8 Updated by Backport Bot 3 months ago

  • Tags set to backport_processed

Also available in: Atom PDF