Project

General

Profile

Bug #53810

memory usage ceiling increased following lua scripts (PR 35477)

Added by Mark Kogan 5 months ago. Updated 8 days ago.

Status:
Fix Under Review
Priority:
High
Target version:
-
% Done:

0%

Source:
Tags:
lua
Backport:
quincy
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

following PR: https://github.com/ceph/ceph/pull/35477 -- rgw/lua: embedding lua in rgw in s3 request context #35477

under specified load and rgw threads/concurrent requests number the real memory rises to 117 GB after 2 hours of load:

while before this pr, memory stays at 48.5 GB overnight of same load

radosgw started as following (note the --rgw_thread_pool_size=4096 --rgw_max_concurrent_requests=8192):

/bin/radosgw --nolockdep -c ./ceph.conf --log-file=./out/radosgw.8000.log --admin-socket=./out/radosgw.8000.asok --pid-file=./out/radosgw.8000.pid -n client.rgw.8000 --rgw_frontends="beast port=8000" --debug_rgw=1 --rgw_thread_pool_size=4096 --rgw_max_concurrent_requests=8192 --rgw_cache_lru_size=1000000 --rgw_max_listing_results=100000 --objecter_inflight_op_bytes=2097152 --objecter_inflight_ops=1024 -f

load generated as following (note the -t 7900 & -z 24M & -n 10000):

while sleep 1; do numactl -N 1 -m 1 -- ~/go/bin/hsbench -a b2345678901234567890 -s b234567890123456789012345678901234567890 -u http://127.0.0.1:8000 -z 24M -d -1 -t 7900 -b $(numactl -N 1 -- nproc) -n 10000 -m g -bp b01b |& tee hsbench.log | stdbuf -oL -eL colrm 180 | ccze -Aonolookups ; done

PS1: when attempting to reproduce ulimit needs to be set accordingly

PS2: commenting the lines in src/rgw/rgw_process.cc prevents the this memory growth from occurring:

...
//auto rc = rgw::lua::read_script(s, store, s->bucket_tenant, s->yield, rgw::lua::context::preRequest, script);
...
//auto rc = rgw::lua::read_script(s, store, s->bucket_tenant, s->yield, rgw::lua::context::postRequest, script);
...

History

#1 Updated by Yuval Lifshitz 5 months ago

  • Assignee set to Yuval Lifshitz
  • Tags set to lua

#2 Updated by Casey Bodley 3 months ago

  • Subject changed from memory usage ceiling increased following PR 35477 to memory usage ceiling increased following lua scripts (PR 35477)

#3 Updated by Casey Bodley 20 days ago

  • Backport set to quincy

#4 Updated by Yuval Lifshitz 8 days ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 46304

Also available in: Atom PDF