Actions
Bug #43055
closedmgr/dashboard: get_rates_from_data, pairwise and thread-unsafe itertools.tee
% Done:
0%
Source:
Q/A
Tags:
Backport:
nautilus
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "traceback": "Traceback (most recent call last):
File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 670, in respond
response.body = self.handler()
File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 220, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/lib/python3.6/site-packages/cherrypy/_cptools.py", line 237, in wrap
return self.newhandler(innerfunc, *args, **kwargs)
File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler
return handler(*args, **kwargs)
File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 60, in __call__
return self.callable(*self.args, **self.kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner
ret = func(*args, **kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 192, in minimal
return self.health_minimal.all_health()
File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 51, in all_health
result['pools'] = self.pools()
File "/usr/share/ceph/mgr/dashboard/controllers/health.py", line 167, in pools
pools = CephService.get_pool_list_with_stats()
File "/usr/share/ceph/mgr/dashboard/services/ceph_service.py", line 124, in get_pool_list_with_stats
'rates': get_rates_from_data(stat_series)
File "/usr/share/ceph/mgr/dashboard/services/ceph_service.py", line 272, in get_rates_from_data
return [(data2[0], differentiate(data1, data2)) for data1, data2 in pairwise(data)]
File "/usr/share/ceph/mgr/dashboard/services/ceph_service.py", line 16, in pairwise
next(b, None)
RuntimeError: deque mutated during iteration
", "version": "8.9.1"}
It seems that this is often caused by multiple threads accessing the iterator (as tee
, invoked by pairwise, is thread-unsafe), but cannot relate any use of multithreading around this. I'd suggest to avoid using pairwise
in get_rates_from_data
, as it can be easily replaced by something less Pythonic but without the risk of using queues (from https://stackoverflow.com/a/20415373):
def pairwise(iterable):
it = iter(iterable)
a = next(it, None)
for b in it:
yield (a, b)
a = b
Updated by Ernesto Puerta about 4 years ago
- Status changed from New to Duplicate
Fixed in: https://tracker.ceph.com/issues/40365
Updated by Ernesto Puerta about 4 years ago
- Is duplicate of Feature #40365: mgr: Add get_rates_from_data from the dashboard to the mgr_util.py added
Updated by Ernesto Puerta about 3 years ago
- Project changed from mgr to Dashboard
- Category changed from 146 to General - Back-end
Actions