Project

General

Profile

Actions

Bug #24768

closed

rgw workload makes osd memory explode

Added by Sage Weil almost 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
mimic,luminous,jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

From ML,

On 07/03/2018 05:55 AM, Sage Weil wrote:
> On Fri, 29 Jun 2018, Aleksei Gutikov wrote:
> > Throughput is 100% the same, just sliced into bigger chunks (rados objects).
> > And this throughput is not high, less than single object per second. And
> > memory stay occupied even after writing stopped.
> > 
> > Currently I'm sure that is side effect of sharing buffer::raw object among
> > different buffer::ptr objects.
> > 
> > Please, have a look into this dump of ObjectContext::attr_cache of one of
> > context in PrimaryLogPG::object_contexts, made after uploading single 4M
> > object into S3.
> > Notice "_user.rgw.idtag" and "_user.rgw.tail_tag" xattrs, both 44 bytes
> > length, holidng 4194304 bytes buffer::raw object (nref=2).
> 
> That is the smoking gun!  What version is this?

Particularly this dump from 12.2.2
But issue was also reproducible for 12.2.5 and master.


Related issues 3 (0 open3 closed)

Copied to RADOS - Backport #24805: mimic: rgw workload makes osd memory explodeResolvedPrashant DActions
Copied to RADOS - Backport #24806: luminous: rgw workload makes osd memory explodeResolvedPrashant DActions
Copied to RADOS - Backport #24847: jewel: rgw workload makes osd memory explodeResolvedKefu ChaiActions
Actions #1

Updated by Sage Weil almost 6 years ago

  • Status changed from 12 to Fix Under Review
Actions #2

Updated by Sage Weil almost 6 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #3

Updated by Nathan Cutler almost 6 years ago

  • Copied to Backport #24805: mimic: rgw workload makes osd memory explode added
Actions #4

Updated by Nathan Cutler almost 6 years ago

  • Copied to Backport #24806: luminous: rgw workload makes osd memory explode added
Actions #5

Updated by Kefu Chai almost 6 years ago

jewel backport: https://github.com/ceph/ceph/pull/22959

i knew that jewel is (almost) EOL. just in case anyone is still stuck with it.

Actions #6

Updated by Nathan Cutler almost 6 years ago

  • Backport changed from mimic,luminous to mimic,luminous,jewel
Actions #7

Updated by Nathan Cutler almost 6 years ago

  • Copied to Backport #24847: jewel: rgw workload makes osd memory explode added
Actions #8

Updated by Kefu Chai over 5 years ago

  • Status changed from Pending Backport to Resolved
Actions

Also available in: Atom PDF