Bug #48358
openrgw: qlen and qactive perf counters leak
0%
Description
In our environment the rgw qlen and qactive perf counters seem to trend slowly upwards. See the plot attached.
I suspect there is a case where the client IO is completed without the qlen/qactive counters getting decremented.
For context, we are trying to see if rgw_max_concurrent_requests can be tuned down to limit the peak rgw memory usage. So we want to monitor how many existing concurrent IOs we have in prod, but clearly this qlen counter isn't reliable for that. We'll send a separate PR to expose the throttle `outstanding_requests` values in a new perf counter to solve this separately, but maybe the qlen leak is obvious to someone?
Files
Updated by Dan van der Ster over 3 years ago
We'll send a separate PR to expose the throttle `outstanding_requests` values in a new perf counter
Updated by Aleksandr Rudenko almost 3 years ago
- File Ceph - RGW metrics - Grafana 2021-05-11 11-27-50.png Ceph - RGW metrics - Grafana 2021-05-11 11-27-50.png added
We can see same behavior on 14.2.15.
Updated by Casey Bodley 3 months ago
- Has duplicate Bug #61338: rgw: qactive perf counter may leak on errors added
Updated by Casey Bodley 3 months ago
- Priority changed from Normal to High
i'm hearing reports that when these counters leak, rgw performance also degrades significantly until the process restarts. this is probably due to leaks of the counter associated with rgw_max_concurrent_requests
to decrement the perf counters, we rely on a call to ClientIO::complete_request(): https://github.com/ceph/ceph/blob/f4758e5/src/rgw/rgw_asio_client.cc#L97-L100
for rgw_max_concurrent_requests, we rely on a similar hook in SimpleThrottler::request_complete(): https://github.com/ceph/ceph/blob/f4758e5/src/rgw/rgw_dmclock_async_scheduler.h#L188-L193
certain types of errors fail to call either function
raising priority since this effects more than just the output of metrics
Updated by Casey Bodley 3 months ago
from Andrea Bolzonella:
After my analysis, I observed that whenever an error is raised in the rgw_rest.cc (line 630 in 18.2.1), the connection is closed, but the qlen is not decremented.
try {
RESTFUL_IO(s)->complete_header();
} catch (rgw::io::Exception& e) {
ldpp_dout(s, 0) << "ERROR: RESTFUL_IO(s)->complete_header() returned err="
<< e.what() << dendl;
}
Updated by Andrea Bolzonella 7 days ago
Is there any progress on this ticket?
We still have a performance issue when active connections get high.