Project

General

Profile

Bug #53810

memory usage ceiling increased following lua scripts (PR 35477)

Added by Mark Kogan about 2 years ago. Updated over 1 year ago.

Status:
Pending Backport
Priority:
High
Target version:
-
% Done:

0%

Source:
Tags:
lua backport_processed
Backport:
quincy
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

following PR: https://github.com/ceph/ceph/pull/35477 -- rgw/lua: embedding lua in rgw in s3 request context #35477

under specified load and rgw threads/concurrent requests number the real memory rises to 117 GB after 2 hours of load:

while before this pr, memory stays at 48.5 GB overnight of same load

radosgw started as following (note the --rgw_thread_pool_size=4096 --rgw_max_concurrent_requests=8192):

/bin/radosgw --nolockdep -c ./ceph.conf --log-file=./out/radosgw.8000.log --admin-socket=./out/radosgw.8000.asok --pid-file=./out/radosgw.8000.pid -n client.rgw.8000 --rgw_frontends="beast port=8000" --debug_rgw=1 --rgw_thread_pool_size=4096 --rgw_max_concurrent_requests=8192 --rgw_cache_lru_size=1000000 --rgw_max_listing_results=100000 --objecter_inflight_op_bytes=2097152 --objecter_inflight_ops=1024 -f

load generated as following (note the -t 7900 & -z 24M & -n 10000):

while sleep 1; do numactl -N 1 -m 1 -- ~/go/bin/hsbench -a b2345678901234567890 -s b234567890123456789012345678901234567890 -u http://127.0.0.1:8000 -z 24M -d -1 -t 7900 -b $(numactl -N 1 -- nproc) -n 10000 -m g -bp b01b |& tee hsbench.log | stdbuf -oL -eL colrm 180 | ccze -Aonolookups ; done

PS1: when attempting to reproduce ulimit needs to be set accordingly

PS2: commenting the lines in src/rgw/rgw_process.cc prevents the this memory growth from occurring:

...
//auto rc = rgw::lua::read_script(s, store, s->bucket_tenant, s->yield, rgw::lua::context::preRequest, script);
...
//auto rc = rgw::lua::read_script(s, store, s->bucket_tenant, s->yield, rgw::lua::context::postRequest, script);
...


Related issues

Copied to rgw - Backport #57179: quincy: memory usage ceiling increased following lua scripts (PR 35477) New

History

#1 Updated by Yuval Lifshitz about 2 years ago

  • Assignee set to Yuval Lifshitz
  • Tags set to lua

#2 Updated by Casey Bodley about 2 years ago

  • Subject changed from memory usage ceiling increased following PR 35477 to memory usage ceiling increased following lua scripts (PR 35477)

#3 Updated by Casey Bodley almost 2 years ago

  • Backport set to quincy

#4 Updated by Yuval Lifshitz almost 2 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 46304

#5 Updated by Mark Kogan almost 2 years ago

quick update, as discussed internally - PR https://github.com/ceph/ceph/pull/46304 does resolve memory creep upwards in overnight test-runs but does not resolve the large (tens of GB) memory jumps

#6 Updated by Casey Bodley over 1 year ago

  • Status changed from Fix Under Review to Pending Backport

#7 Updated by Backport Bot over 1 year ago

  • Copied to Backport #57179: quincy: memory usage ceiling increased following lua scripts (PR 35477) added

#8 Updated by Backport Bot over 1 year ago

  • Tags changed from lua to lua backport_processed

Also available in: Atom PDF