Project

General

Profile

Bug #24768

rgw workload makes osd memory explode

Added by Sage Weil about 1 year ago. Updated 11 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
Start date:
07/04/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
mimic,luminous,jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:

Description

From ML,

On 07/03/2018 05:55 AM, Sage Weil wrote:
> On Fri, 29 Jun 2018, Aleksei Gutikov wrote:
> > Throughput is 100% the same, just sliced into bigger chunks (rados objects).
> > And this throughput is not high, less than single object per second. And
> > memory stay occupied even after writing stopped.
> > 
> > Currently I'm sure that is side effect of sharing buffer::raw object among
> > different buffer::ptr objects.
> > 
> > Please, have a look into this dump of ObjectContext::attr_cache of one of
> > context in PrimaryLogPG::object_contexts, made after uploading single 4M
> > object into S3.
> > Notice "_user.rgw.idtag" and "_user.rgw.tail_tag" xattrs, both 44 bytes
> > length, holidng 4194304 bytes buffer::raw object (nref=2).
> 
> That is the smoking gun!  What version is this?

Particularly this dump from 12.2.2
But issue was also reproducible for 12.2.5 and master.


Related issues

Copied to RADOS - Backport #24805: mimic: rgw workload makes osd memory explode Resolved
Copied to RADOS - Backport #24806: luminous: rgw workload makes osd memory explode Resolved
Copied to RADOS - Backport #24847: jewel: rgw workload makes osd memory explode Resolved

History

#1 Updated by Sage Weil about 1 year ago

  • Status changed from Verified to Need Review

#2 Updated by Sage Weil about 1 year ago

  • Status changed from Need Review to Pending Backport

#3 Updated by Nathan Cutler about 1 year ago

  • Copied to Backport #24805: mimic: rgw workload makes osd memory explode added

#4 Updated by Nathan Cutler about 1 year ago

  • Copied to Backport #24806: luminous: rgw workload makes osd memory explode added

#5 Updated by Kefu Chai about 1 year ago

jewel backport: https://github.com/ceph/ceph/pull/22959

i knew that jewel is (almost) EOL. just in case anyone is still stuck with it.

#6 Updated by Nathan Cutler about 1 year ago

  • Backport changed from mimic,luminous to mimic,luminous,jewel

#7 Updated by Nathan Cutler about 1 year ago

  • Copied to Backport #24847: jewel: rgw workload makes osd memory explode added

#8 Updated by Kefu Chai 11 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF