Project

General

Profile

Bug #22410

rgw: chained cache size is growing above rgw_cache_lru_size limit

Added by Mark Kogan over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
Start date:
12/12/2017
Due date:
% Done:

0%

Source:
Development
Tags:
Backport:
jewel luminous
Regression:
No
Severity:
2 - major
Reviewed:
ceph-qa-suite:
Pull request ID:

Description

Details and analysis:

valgrind massif shown the following information:

--------------------------------------------------------------------------------              
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)               
--------------------------------------------------------------------------------              
 65 4,708,134,638,372    2,339,301,056    2,141,918,346   197,382,710            0            
91.56% (2,141,918,346B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.       
->37.62% (880,128,000B) 0x4A439E: RGWChainedCacheImpl<bucket_info_entry>::chain_cb(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void*) (new_alloca
| ->37.62% (880,128,000B) 0x5C13DF: ObjectCache::chain_cache_entry(std::__cxx11::list<rgw_cache_entry_info*, std::allocator<rgw_cache_entry_info*> >&, RGWChainedCache::Entry*) (rgw_cache.cc
|   ->37.62% (880,128,000B) 0x4486DC: RGWRados::get_bucket_info(RGWObjectCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_st...

Added a log in chain_cb(() (rgw_rados.h)

void chain_cb(const string& key, void *data) override {
    T *entry = static_cast<T *>(data);
    RWLock::WLocker wl(lock);

    lsubdout(g_ceph_context, rgw, 10)
        << "#DEBUG# " << __func__ << " key='" << key << "' entries.size()=" << entries.size() << dendl;

    entries[key].first = *entry;
    if (expiry.count() > 0) {
      entries[key].second = ceph::coarse_mono_clock::now();
    }
  }

And saw that the entries.size() was increasing above _conf->rgw_cache_lru_size (10000)

2017-12-12 03:38:46.426 4a5d1700 10 #DEBUG# chain_cb key='mycontainers754053' entries.size()=754053


Related issues

Copied to rgw - Backport #22583: luminous: rgw: chained cache size is growing above rgw_cache_lru_size limit Resolved
Copied to rgw - Backport #22584: jewel: rgw: chained cache size is growing above rgw_cache_lru_size limit Resolved

History

#1 Updated by Mark Kogan over 1 year ago

  • Description updated (diff)

#2 Updated by Matt Benjamin over 1 year ago

  • Status changed from New to Testing
  • Priority changed from Urgent to High
  • Backport set to jewel luminous

#3 Updated by Yehuda Sadeh over 1 year ago

  • Subject changed from Chained cache size is growing above rgw_cache_lru_size limit to rgw: chained cache size is growing above rgw_cache_lru_size limit

#4 Updated by Ken Dreyer over 1 year ago

  • Status changed from Testing to Pending Backport

#5 Updated by Nathan Cutler over 1 year ago

  • Copied to Backport #22583: luminous: rgw: chained cache size is growing above rgw_cache_lru_size limit added

#6 Updated by Nathan Cutler over 1 year ago

  • Copied to Backport #22584: jewel: rgw: chained cache size is growing above rgw_cache_lru_size limit added

#7 Updated by Yuri Weinstein over 1 year ago

https://github.com/ceph/ceph/pull/19785 (luminous backport - see #22583) was merged

#8 Updated by Nathan Cutler over 1 year ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF