Bug #50205
openTelemetry module failing on empty JSON
0%
Description
Ubuntu 16.04, 18.04
Ceph 14.2.11+
Telemetry module is not validating JSON while trying to anonymize hard drives serail numbers:
2021-04-06 10:05:47.671 7f783a83c700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'telemetry' while running on mgr.rgw3: No JSON object could be decoded 2021-04-06 10:05:47.671 7f783a83c700 -1 telemetry.serve: 2021-04-06 10:05:47.671 7f783a83c700 -1 Traceback (most recent call last): File "/usr/share/ceph/mgr/telemetry/module.py", line 832, in serve self.send(self.last_report) File "/usr/share/ceph/mgr/telemetry/module.py", line 716, in send devices = self.gather_device_report() File "/usr/share/ceph/mgr/telemetry/module.py", line 414, in gather_device_report m = json.loads(m_str.replace(serial, 'deleted')) File "/usr/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded
Updated by Neha Ojha about 3 years ago
- Assignee set to Yaarit Hatuka
Yaarit, could you please take a look?
Updated by Michał Chybowski about 3 years ago
This issue should be fixed with PR https://github.com/ceph/ceph/pull/40669
Updated by Yaarit Hatuka about 3 years ago
- Status changed from New to Fix Under Review
- Pull request ID set to 40669
Updated by Yaarit Hatuka about 3 years ago
Hi Michał,
Many thanks for your fix!
I see the issue you’re facing happens in version 14.2.11:
https://github.com/ceph/ceph/blob/v14.2.11/src/pybind/mgr/telemetry/module.py#L412-L414
and we had a fix for it, which was backported to version 14.2.12:
https://github.com/ceph/ceph/commit/0dccfc4196900e2f9e03e6d2c4ab3edcc46023aa
The problem seems to be that ‘serial’ itself is empty, and when we try to replace the empty string in the serialized map ('m_str') with ‘deleted’, we get a string which cannot be loaded to JSON:
https://github.com/ceph/ceph/blob/v14.2.11/src/pybind/mgr/telemetry/module.py#L414
Even if ‘m_str’ itself was empty, but ‘serial’ was valid - we wouldn’t see this error, since
m_str.replace(serial, 'deleted')would return an empty string, which could be loaded to JSON.
Can you please try to apply this fix and see that it’s working?
Thanks!
Yaarit
Updated by Loïc Dachary about 3 years ago
- Target version changed from v14.2.20 to v14.2.21
Updated by Neha Ojha almost 3 years ago
- Status changed from Fix Under Review to Need More Info