Bug #8289
closedrgw: memory not freed during in-progress read (dumpling)
0%
Description
observing memory growing linearly during a read on a large object. once the read completes, memory is freed.tcmalloc
profiling indicates it is all allocated during read_message(). it looks like we are keeping a ref to the incoming buffers somewhere while streaming the incoming data.
Updated by Sage Weil almost 10 years ago
workaround is to limit the number of threads so that only a handful of such large reads can be in progress at once; this prevents the box from OOMing.
ceph daemon /var/run/ceph/*radosgw*asok config set rgw_thread_pool_size 2or similar
Updated by Sage Weil almost 10 years ago
in this particular case the request is getting 503 after 30 seconds (the fastcgi timeout?).
Updated by Greg Farnum almost 10 years ago
Do we know if the object is a multi-part upload or not? And is it chunked or a single RADOS object? This might only be a problem for legacy systems. crosses fingers
Updated by Yehuda Sadeh almost 10 years ago
I tend to think that what we see is that we read the data off RADOS much quicker than the client reading the data back, and with the (relatively recent) fix that pushes the data to the client outside of the callback thread we end up accumulating all the new data without any throttling. I'll need to verify that though.
Updated by Yehuda Sadeh almost 10 years ago
Could very well be related to #8269, in which we don't handle nicely the cancellation.
Updated by Ian Colle almost 10 years ago
- Priority changed from Immediate to Urgent
Updated by Ian Colle almost 10 years ago
- Status changed from 12 to In Progress
- Assignee set to Yehuda Sadeh
Updated by Yehuda Sadeh almost 10 years ago
- Status changed from In Progress to Fix Under Review
Updated by Yehuda Sadeh almost 10 years ago
- Assignee changed from Yehuda Sadeh to Josh Durgin
Updated by Sage Weil almost 10 years ago
- Status changed from Fix Under Review to Resolved