Ceph » mgr » Dashboard

{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "traceback": "Traceback (most recent call last):
  File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 670, in respond
    response.body = self.handler()
  File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 220, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/lib/python3.6/site-packages/cherrypy/_cptools.py", line 237, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 60, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 192, in minimal
    return self.health_minimal.all_health()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 51, in all_health
    result['pools'] = self.pools()
  File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 167, in pools
    pools = CephService.get_pool_list_with_stats()
  File "/usr/share/ceph/mgr/dashboard/services/ceph_service.py", line 124, in get_pool_list_with_stats
    'rates': get_rates_from_data(stat_series)
  File "/usr/share/ceph/mgr/dashboard/services/ceph_service.py", line 272, in get_rates_from_data
    return [(data2[0], differentiate(data1, data2)) for data1, data2 in pairwise(data)]
  File "/usr/share/ceph/mgr/dashboard/services/ceph_service.py", line 16, in pairwise
    next(b, None)
RuntimeError: deque mutated during iteration
", "version": "8.9.1"}

It seems that this is often caused by multiple threads accessing the iterator (as tee, invoked by pairwise, is thread-unsafe), but cannot relate any use of multithreading around this. I'd suggest to avoid using pairwise in get_rates_from_data, as it can be easily replaced by something less Pythonic but without the risk of using queues (from https://stackoverflow.com/a/20415373):

def pairwise(iterable):
    it = iter(iterable)
    a = next(it, None)

    for b in it:
        yield (a, b)
        a = b