Bug #53603: mgr/telemetry: list index out of range in gather_device_report - mgr - Ceph

Custom queries

Backports
Backports: nautilus
Bug queue
Bug triage
Crash queue
Crash triage
Dashboard Telemetry issues
Feedback
Low Hanging Fruit
My issues
Need Review
New Dashboard Bugs
Orchestrator
Pending backports
Product Backlog Scrub

Actions

Copy link

Bug #53603

closed

mgr/telemetry: list index out of range in gather_device_report

Added by Yaarit Hatuka over 2 years ago. Updated about 1 year ago.

Status:

Resolved

Priority:

Normal

Assignee:

Yaarit Hatuka

Category:

telemetry module

Target version:

Ceph - v17.0.0

% Done:

Source:

Development

Tags:

backport_processed

Backport:

pacific, octopus

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v15.0.0, Ceph - v15.2.1, Ceph - v15.2.10, Ceph - v15.2.11, Ceph - v15.2.12, Ceph - v15.2.13, Ceph - v15.2.14, Ceph - v15.2.15, Ceph - v15.2.16, Ceph - v15.2.2, Ceph - v15.2.3, Ceph - v15.2.4, Ceph - v15.2.5, Ceph - v15.2.6, Ceph - v15.2.7, Ceph - v15.2.8, Ceph - v15.2.9, Ceph - v16.0.0, Ceph - v16.0.1, Ceph - v16.1.0, Ceph - v16.1.1, Ceph - v16.2.0, Ceph - v16.2.1, Ceph - v16.2.2, Ceph - v16.2.3, Ceph - v16.2.4, Ceph - v16.2.5, Ceph - v16.2.6, Ceph - v16.2.7, Ceph - v16.2.8, Ceph - v17.0.0

ceph-qa-suite:

Pull request ID:

44327

Crash signature (v1):

Crash signature (v2):

Description

Telemetry crashed on the gibba cluster:

# ceph health detail
HEALTH_ERR Module 'telemetry' has failed: list index out of range; 3 daemons have recently crashed
[ERR] MGR_MODULE_ERROR: Module 'telemetry' has failed: list index out of range
    Module 'telemetry' has failed: list index out of range

2021-12-14T16:44:38.413+0000 7f3372dd4700 -1 Traceback (most recent call last):
  File "/usr/share/ceph/mgr/telemetry/module.py", line 1152, in serve
    self.send(self.last_report)
  File "/usr/share/ceph/mgr/telemetry/module.py", line 963, in send
    devices = self.gather_device_report()
  File "/usr/share/ceph/mgr/telemetry/module.py", line 611, in gather_device_report
    host = d['location'][0]['host']
IndexError: list index out of range

Related issues 3 (0 open — 3 closed)

Related to mgr - Bug #53604: mgr/telemetry: list assignment index out of range in gather_crashinfo

Resolved

Yaarit Hatuka

Actions

Copied to mgr - Backport #53691: pacific: mgr/telemetry: list index out of range in gather_device_report

Rejected

Actions

Copied to mgr - Backport #53692: octopus: mgr/telemetry: list index out of range in gather_device_report

Rejected

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by Vikhyat Umrao over 2 years ago

2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr dispatch_remote Success calling 'ls'
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr ~Gil Destroying new thread state 0x55eb2c4a9600

2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr Gil Switched to new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr ~Gil Destroying new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr Gil Switched to new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr dispatch_remote Calling crash.do_info...
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr dispatch_remote Success calling 'do_info'
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr ~Gil Destroying new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr Gil Switched to new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr ~Gil Destroying new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr Gil Switched to new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr dispatch_remote Calling crash.do_info...
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr dispatch_remote Success calling 'do_info'
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr ~Gil Destroying new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr Gil Switched to new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr ~Gil Destroying new thread state 0x55eb2c4a9600

2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr Gil Switched to new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr dispatch_remote Calling crash.do_info...
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr dispatch_remote Success calling 'do_info'
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr ~Gil Destroying new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr Gil Switched to new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr ~Gil Destroying new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr Gil Switched to new thread state 0x55eb2c4a9600
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr dispatch_remote Calling crash.do_info...

2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr dispatch_remote Success calling 'do_info'
2021-12-14T18:48:04.034+0000 7ffa856af700 20 mgr ~Gil Destroying new thread state 0x55eb2c4a9600

2021-12-14T18:48:04.034+0000 7ffa856af700 -1 mgr handle_command module 'telemetry' command handler threw exception: list assignment index out of range

2021-12-14T18:48:04.116+0000 7ffa856af700 20 mgr ~Gil Destroying new thread state 0x55eb061b3b00
2021-12-14T18:48:04.116+0000 7ffa856af700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1648, in _handle_command
    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 434, in call
    return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/telemetry/module.py", line 1038, in show
    report = self.get_report(channels=channels)
  File "/usr/share/ceph/mgr/telemetry/module.py", line 1087, in get_report
    return self.compile_report(channels=channels)
  File "/usr/share/ceph/mgr/telemetry/module.py", line 900, in compile_report
    report['crashes'] = self.gather_crashinfo()
  File "/usr/share/ceph/mgr/telemetry/module.py", line 483, in gather_crashinfo
    c['backtrace'][-1] = '<redacted>'
IndexError: list assignment index out of range

2021-12-14T18:48:04.116+0000 7ffa856af700  1 -- [v2:172.21.2.101:7002/1319847290,v1:172.21.2.101:7003/1319847290] --> 172.21.2.102:0/1267436291 -- mgr_command_reply(tid 0: -22 Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1648, in _handle_command
    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 434, in call
    return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/telemetry/module.py", line 1038, in show
    report = self.get_report(channels=channels)
  File "/usr/share/ceph/mgr/telemetry/module.py", line 1087, in get_report
    return self.compile_report(channels=channels)
  File "/usr/share/ceph/mgr/telemetry/module.py", line 900, in compile_report
    report['crashes'] = self.gather_crashinfo()
  File "/usr/share/ceph/mgr/telemetry/module.py", line 483, in gather_crashinfo
    c['backtrace'][-1] = '<redacted>'
IndexError: list assignment index out of range
) v1 -- 0x55eb10343a20 con 0x55eb03927800
2021-12-14T18:48:04.116+0000 7ffa856af700 10 mgr.server operator()  command returned -22
2021-12-14T18:48:04.116+0000 7ffa856af700 20 mgr Gil Switched to new thread state 0x55eb061b3b00
2021-12-14T18:48:04.116+0000 7ffa856af700 20 mgr ~Gil Destroying new thread state 0x55eb061b3b00

Actions

Copy link