Bug #53810
memory usage ceiling increased following lua scripts (PR 35477)
0%
Description
following PR: https://github.com/ceph/ceph/pull/35477 -- rgw/lua: embedding lua in rgw in s3 request context #35477
under specified load and rgw threads/concurrent requests number the real memory rises to 117 GB after 2 hours of load:
while before this pr, memory stays at 48.5 GB overnight of same load
radosgw started as following (note the --rgw_thread_pool_size=4096 --rgw_max_concurrent_requests=8192):
/bin/radosgw --nolockdep -c ./ceph.conf --log-file=./out/radosgw.8000.log --admin-socket=./out/radosgw.8000.asok --pid-file=./out/radosgw.8000.pid -n client.rgw.8000 --rgw_frontends="beast port=8000" --debug_rgw=1 --rgw_thread_pool_size=4096 --rgw_max_concurrent_requests=8192 --rgw_cache_lru_size=1000000 --rgw_max_listing_results=100000 --objecter_inflight_op_bytes=2097152 --objecter_inflight_ops=1024 -f
load generated as following (note the -t 7900 & -z 24M & -n 10000):
while sleep 1; do numactl -N 1 -m 1 -- ~/go/bin/hsbench -a b2345678901234567890 -s b234567890123456789012345678901234567890 -u http://127.0.0.1:8000 -z 24M -d -1 -t 7900 -b $(numactl -N 1 -- nproc) -n 10000 -m g -bp b01b |& tee hsbench.log | stdbuf -oL -eL colrm 180 | ccze -Aonolookups ; done
PS1: when attempting to reproduce ulimit needs to be set accordingly
PS2: commenting the lines in src/rgw/rgw_process.cc prevents the this memory growth from occurring:
... //auto rc = rgw::lua::read_script(s, store, s->bucket_tenant, s->yield, rgw::lua::context::preRequest, script); ... //auto rc = rgw::lua::read_script(s, store, s->bucket_tenant, s->yield, rgw::lua::context::postRequest, script); ...
Related issues
History
#1 Updated by Yuval Lifshitz about 2 years ago
- Assignee set to Yuval Lifshitz
- Tags set to lua
#2 Updated by Casey Bodley about 2 years ago
- Subject changed from memory usage ceiling increased following PR 35477 to memory usage ceiling increased following lua scripts (PR 35477)
#3 Updated by Casey Bodley almost 2 years ago
- Backport set to quincy
#4 Updated by Yuval Lifshitz almost 2 years ago
- Status changed from New to Fix Under Review
- Pull request ID set to 46304
#5 Updated by Mark Kogan almost 2 years ago
quick update, as discussed internally - PR https://github.com/ceph/ceph/pull/46304 does resolve memory creep upwards in overnight test-runs but does not resolve the large (tens of GB) memory jumps
#6 Updated by Casey Bodley over 1 year ago
- Status changed from Fix Under Review to Pending Backport
#7 Updated by Backport Bot over 1 year ago
- Copied to Backport #57179: quincy: memory usage ceiling increased following lua scripts (PR 35477) added
#8 Updated by Backport Bot over 1 year ago
- Tags changed from lua to lua backport_processed