Bug #23207
rgw: inefficient buffer usage for PUTs
0%
Description
At least in jewel using swift, radosgw is very inefficient about its buffering process. In RGWPutObj_ObjStore::get_data(), it allocates a buffer usually of size rgw_max_chunk_size, issues a single "mg_read()" call (which might fill about 1% of the buffer), then passes the result along. Eventually a list of buffers is condensed and given to write, but meanwhile there is a lot of wasted memory. With a chunk size of 1mb and a workload that wrote and deleted 262 objects using 64 parallel threads where no object was > 33m (avg size 7m), I was finding a peak allocation of 3.7g of ram, of which 99% was never being used.
I have an experimental patch that fixes this behavior. With that patch, using the same workload, I am now usually filling 1m buffers, the buffers do not need to be repacked to write them, and peak allocation went down to 138m of ram.
Related issues
History
#1 Updated by Vikhyat Umrao about 6 years ago
- Status changed from New to In Progress
#2 Updated by Marcus Watts about 6 years ago
I updated my PR. I moved the fill loop down from just the put-object code path to just above mg_read(). I don't believe there was logic to handle short-reads anywhere else, and we've gotten reports of hard to reproduce signature verification failures that could be explained by short reads, if so that should be fixed with this updated patch.
It's likely main | luminous have the same problem but I have not yet looked to see if that's the case.
#3 Updated by Matt Benjamin about 6 years ago
great work, thanks marcus!
Matt
#4 Updated by Marcus Watts about 6 years ago
I checked a copy of master. It does not have the problem. I'll need to do more checking; this is a mutant master linked against openssl 1.1.
#5 Updated by Marcus Watts about 6 years ago
Matt suggested that it might be best to apply this fix to master - that way it doesn't matter what behavior a particular version of civetweb has; our code will work with it. So I've put together,
https://github.com/ceph/ceph/pull/20724
#6 Updated by Nathan Cutler about 6 years ago
- Backport set to luminous, jewel
#7 Updated by Yehuda Sadeh about 6 years ago
- Status changed from In Progress to Fix Under Review
#8 Updated by Casey Bodley about 6 years ago
- Subject changed from rgw: (jewel) inefficient buffer usage for PUTs to rgw: inefficient buffer usage for PUTs
- Status changed from Fix Under Review to Pending Backport
#9 Updated by Nathan Cutler about 6 years ago
- Copied to Backport #23347: luminous: rgw: inefficient buffer usage for PUTs added
#10 Updated by Nathan Cutler about 6 years ago
- Copied to Backport #23348: jewel: rgw: inefficient buffer usage for PUTs added
#11 Updated by Nathan Cutler almost 6 years ago
- Status changed from Pending Backport to Resolved
#12 Updated by Nathan Cutler almost 6 years ago
- Related to Bug #23596: mg_read() call has wrong arguments added