Bug #8289
closed
rgw: memory not freed during in-progress read (dumpling)
Added by Sage Weil almost 10 years ago.
Updated almost 10 years ago.
Description
observing memory growing linearly during a read on a large object. once the read completes, memory is freed.tcmalloc
profiling indicates it is all allocated during read_message(). it looks like we are keeping a ref to the incoming buffers somewhere while streaming the incoming data.
workaround is to limit the number of threads so that only a handful of such large reads can be in progress at once; this prevents the box from OOMing.
ceph daemon /var/run/ceph/*radosgw*asok config set rgw_thread_pool_size 2
or similar
in this particular case the request is getting 503 after 30 seconds (the fastcgi timeout?).
Do we know if the object is a multi-part upload or not? And is it chunked or a single RADOS object? This might only be a problem for legacy systems. crosses fingers
I tend to think that what we see is that we read the data off RADOS much quicker than the client reading the data back, and with the (relatively recent) fix that pushes the data to the client outside of the callback thread we end up accumulating all the new data without any throttling. I'll need to verify that though.
Could very well be related to #8269, in which we don't handle nicely the cancellation.
- Priority changed from Immediate to Urgent
- Status changed from 12 to In Progress
- Assignee set to Yehuda Sadeh
- Status changed from In Progress to Fix Under Review
- Assignee changed from Yehuda Sadeh to Josh Durgin
- Status changed from Fix Under Review to Resolved
Also available in: Atom
PDF