Bug #53538: mgr/stats: ZeroDivisionError - mgr - Ceph

[ubuntu@gibba001 ~]$ sudo ceph osd status|grep gibba043|wc -l
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1648, in _handle_command
    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 434, in call
    return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/status/module.py", line 338, in handle_osd_status
    wr_ops_rate = (self.get_rate("osd", osd_id.__str__(), "osd.op_w") +
  File "/usr/share/ceph/mgr/status/module.py", line 28, in get_rate
    return (data[-1][1] - data[-2][1]) // int(data[-1][0] - data[-2][0])
ZeroDivisionError: integer division or modulo by zero

Actions

Copy link

#5

Updated by Neha Ojha over 2 years ago

Assignee set to Nitzan Mordechai

Actions

Copy link

#6

Updated by Nitzan Mordechai over 2 years ago

From my understanding, the status update interval by mgr_stats_period configuration value, the division is failing when that statement int(data[-1]⁰ - data[-2]⁰) = 0 , data[-1]⁰ and data[-2]⁰ holding timestamps of the 2 last stats that were updated, to get 0 in that Subtraction - (data[-1]⁰ == data[-2]⁰) or (data[-1]⁰ - data[-2]⁰ < 1) we are checking only for the first condition and both cases can only happen when we update the stats fast enough - that means we had mgr_stats_period = 1 or something else caused the stats to be updated in range of less then 1 second.

Actions

Copy link

#7

Updated by Sebastian Wagner over 2 years ago

Nitzan Mordechai wrote:

From my understanding, the status update interval by mgr_stats_period configuration value, the division is failing when that statement int(data[-1]⁰ - data[-2]⁰) = 0 , data[-1]⁰ and data[-2]⁰ holding timestamps of the 2 last stats that were updated, to get 0 in that Subtraction - (data[-1]⁰ == data[-2]⁰) or (data[-1]⁰ - data[-2]⁰ < 1) we are checking only for the first condition and both cases can only happen when we update the stats fast enough - that means we had mgr_stats_period = 1 or something else caused the stats to be updated in range of less then 1 second.

Yes and we had this is problem in the dashbaord already. Our solution was https://github.com/ceph/ceph/pull/28603 . If you just use the same function also in the stats module, things should work properly

Actions

Copy link

#8