Ceph » rgw

At least in jewel using swift, radosgw is very inefficient about its buffering process. In RGWPutObj_ObjStore::get_data(), it allocates a buffer usually of size rgw_max_chunk_size, issues a single "mg_read()" call (which might fill about 1% of the buffer), then passes the result along. Eventually a list of buffers is condensed and given to write, but meanwhile there is a lot of wasted memory. With a chunk size of 1mb and a workload that wrote and deleted 262 objects using 64 parallel threads where no object was > 33m (avg size 7m), I was finding a peak allocation of 3.7g of ram, of which 99% was never being used.

I have an experimental patch that fixes this behavior. With that patch, using the same workload, I am now usually filling 1m buffers, the buffers do not need to be repacked to write them, and peak allocation went down to 138m of ram.