rgw: inefficient buffer usage for PUTs
At least in jewel using swift, radosgw is very inefficient about its buffering process. In RGWPutObj_ObjStore::get_data(), it allocates a buffer usually of size rgw_max_chunk_size, issues a single "mg_read()" call (which might fill about 1% of the buffer), then passes the result along. Eventually a list of buffers is condensed and given to write, but meanwhile there is a lot of wasted memory. With a chunk size of 1mb and a workload that wrote and deleted 262 objects using 64 parallel threads where no object was > 33m (avg size 7m), I was finding a peak allocation of 3.7g of ram, of which 99% was never being used.
I have an experimental patch that fixes this behavior. With that patch, using the same workload, I am now usually filling 1m buffers, the buffers do not need to be repacked to write them, and peak allocation went down to 138m of ram.
#2 Updated by Marcus Watts over 2 years ago
I updated my PR. I moved the fill loop down from just the put-object code path to just above mg_read(). I don't believe there was logic to handle short-reads anywhere else, and we've gotten reports of hard to reproduce signature verification failures that could be explained by short reads, if so that should be fixed with this updated patch.
It's likely main | luminous have the same problem but I have not yet looked to see if that's the case.