Project

General

Profile

Bug #50205

Telemetry module failing on empty JSON

Added by Michał Chybowski 14 days ago. Updated about 8 hours ago.


Description

Ubuntu 16.04, 18.04

Ceph 14.2.11+

Telemetry module is not validating JSON while trying to anonymize hard drives serail numbers:

2021-04-06 10:05:47.671 7f783a83c700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'telemetry' while running on mgr.rgw3: No JSON object could be decoded
2021-04-06 10:05:47.671 7f783a83c700 -1 telemetry.serve:
2021-04-06 10:05:47.671 7f783a83c700 -1 Traceback (most recent call last):
  File "/usr/share/ceph/mgr/telemetry/module.py", line 832, in serve
    self.send(self.last_report)
  File "/usr/share/ceph/mgr/telemetry/module.py", line 716, in send
    devices = self.gather_device_report()
  File "/usr/share/ceph/mgr/telemetry/module.py", line 414, in gather_device_report
    m = json.loads(m_str.replace(serial, 'deleted'))
  File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

History

#1 Updated by Neha Ojha 12 days ago

  • Assignee set to Yaarit Hatuka

Yaarit, could you please take a look?

#2 Updated by Michał Chybowski 11 days ago

This issue should be fixed with PR https://github.com/ceph/ceph/pull/40669

#3 Updated by Yaarit Hatuka 9 days ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 40669

#4 Updated by Yaarit Hatuka 9 days ago

Hi Michał,

Many thanks for your fix!

I see the issue you’re facing happens in version 14.2.11:
https://github.com/ceph/ceph/blob/v14.2.11/src/pybind/mgr/telemetry/module.py#L412-L414

and we had a fix for it, which was backported to version 14.2.12:
https://github.com/ceph/ceph/commit/0dccfc4196900e2f9e03e6d2c4ab3edcc46023aa

The problem seems to be that ‘serial’ itself is empty, and when we try to replace the empty string in the serialized map ('m_str') with ‘deleted’, we get a string which cannot be loaded to JSON:
https://github.com/ceph/ceph/blob/v14.2.11/src/pybind/mgr/telemetry/module.py#L414

Even if ‘m_str’ itself was empty, but ‘serial’ was valid - we wouldn’t see this error, since

m_str.replace(serial, 'deleted')
would return an empty string, which could be loaded to JSON.

Can you please try to apply this fix and see that it’s working?

Thanks!
Yaarit

#5 Updated by Loïc Dachary about 8 hours ago

  • Target version changed from v14.2.20 to v14.2.21

Also available in: Atom PDF