Project

General

Profile

Bug #7385

Objectcacher setting max object counts too low

Added by Mark Nelson over 9 years ago. Updated about 8 years ago.

Status:
Resolved
Priority:
High
Assignee:
Jason Dillaman
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
hammer,firefly
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

It appears that the objectcacher is setting max object counts based on the max dirty data size and object size. With default RBD block size and RBD cache values, this was max objects = 42 according to objectcacher logs. In performance testing performed on our new test hardware, this caused severe rbd cache thrashing with high amounts of concurrent IO (8 threads with io_depth = 16 each). With lower amounts of concurrent IO, journal writes on the OSDs were on average around 400-450K in size (with a default 512 max_sectors_kb value). With higher concurrent IOs, this decreased to around 12-16K default journal write size and significantly lower performance.

This may also explain the odd graphs that we saw in our cuttlefish testing with multiple volumes on a QEMU/KVM guest as the io depth increased:

http://ceph.com/wp-content/uploads/2014/07/cuttlefish-rbd_btrfs-write-0004K.png

This can likely be worked around in current code by increasing the rbd dirty cache limits, but ultimately may be fixed by changing the behavior for how object limits are set in the objectcacher.

Related issues

Copied to rbd - Backport #11730: Objectcacher setting max object counts too low Resolved 02/10/2014
Copied to rbd - Backport #11731: Objectcacher setting max object counts too low Resolved 02/10/2014

Associated revisions

Revision 0559d31d (diff)
Added by Josh Durgin over 9 years ago

librbd: remove limit on number of objects in the cache

The number of objects is not a significant indicated of when data
should be written out for rbd. Use the highest possible value for
number of objects and just rely on the dirty data limits to trigger
flushing. When the number of objects is low, and many start being
flushed before they accumulate many requests, it hurts average request
size and performance for many concurrent sequential writes.

Fixes: #7385
Backport: emperor, dumpling
Signed-off-by: Josh Durgin <>

Revision 48a71e22 (diff)
Added by Josh Durgin over 9 years ago

librbd: remove limit on number of objects in the cache

The number of objects is not a significant indicated of when data
should be written out for rbd. Use the highest possible value for
number of objects and just rely on the dirty data limits to trigger
flushing. When the number of objects is low, and many start being
flushed before they accumulate many requests, it hurts average request
size and performance for many concurrent sequential writes.

Fixes: #7385
Backport: emperor, dumpling
Signed-off-by: Josh Durgin <>
(cherry picked from commit 0559d31db29ea83bdb6cec72b830d16b44e3cd35)

Revision 367cf1bb (diff)
Added by Josh Durgin over 9 years ago

librbd: remove limit on number of objects in the cache

The number of objects is not a significant indicated of when data
should be written out for rbd. Use the highest possible value for
number of objects and just rely on the dirty data limits to trigger
flushing. When the number of objects is low, and many start being
flushed before they accumulate many requests, it hurts average request
size and performance for many concurrent sequential writes.

Fixes: #7385
Backport: emperor, dumpling
Signed-off-by: Josh Durgin <>
(cherry picked from commit 0559d31db29ea83bdb6cec72b830d16b44e3cd35)

Revision 0b378942 (diff)
Added by Jason Dillaman over 8 years ago

librbd: updated cache max objects calculation

The previous calculation was based upon the image's object size.
Since the cache stores smaller bufferheads, the object size is not
a good indicator of cache usage and was resulting in objects being
evicted from the cache too often. Instead, base the max number of
objects on the memory load required to store the extra metadata
for the objects.

Fixes: #7385
Backport: firefly, hammer
Signed-off-by: Jason Dillaman <>

Revision 2864da86 (diff)
Added by Jason Dillaman over 8 years ago

librbd: updated cache max objects calculation

The previous calculation was based upon the image's object size.
Since the cache stores smaller bufferheads, the object size is not
a good indicator of cache usage and was resulting in objects being
evicted from the cache too often. Instead, base the max number of
objects on the memory load required to store the extra metadata
for the objects.

Fixes: #7385
Backport: firefly, hammer
Signed-off-by: Jason Dillaman <>
(cherry picked from commit 0b378942c4f1b79cb65967f2d3466728ca1c8d5b)

Conflicts:
src/librbd/ImageCtx.cc
because hammer has cct->_conf->rbd_cache_size
instead of cache_size

Revision 9845258f (diff)
Added by Jason Dillaman over 8 years ago

librbd: updated cache max objects calculation

The previous calculation was based upon the image's object size.
Since the cache stores smaller bufferheads, the object size is not
a good indicator of cache usage and was resulting in objects being
evicted from the cache too often. Instead, base the max number of
objects on the memory load required to store the extra metadata
for the objects.

Fixes: #7385
Backport: firefly, hammer
Signed-off-by: Jason Dillaman <>
(cherry picked from commit 0b378942c4f1b79cb65967f2d3466728ca1c8d5b)

Conflicts:
src/librbd/ImageCtx.cc
because firefly has cct->_conf->rbd_cache_size
instead of cache_size

History

#1 Updated by Josh Durgin over 9 years ago

  • Category changed from ObjectCacher to librbd

This is set by librbd or ceph-fuse's Client.cc after creating the objectcacher. There's already a config option for Client.cc, so just librbd needs fixing.

#2 Updated by Mark Nelson over 9 years ago

Does it make sense to be exposing this to librbd/client.cc at all vs just directly setting it via a config option? (or even eliminating it?)

#3 Updated by Sage Weil over 9 years ago

  • Priority changed from Normal to Urgent

#4 Updated by Ian Colle over 9 years ago

  • Assignee set to David Zafman

#5 Updated by Josh Durgin over 9 years ago

  • Status changed from New to Fix Under Review
  • Assignee changed from David Zafman to Josh Durgin
  • Backport set to emperor, dumpling

#6 Updated by Sage Weil over 9 years ago

  • Status changed from Fix Under Review to Pending Backport

#7 Updated by Sage Weil over 9 years ago

  • Status changed from Pending Backport to 12

#8 Updated by Sage Weil over 9 years ago

  • Status changed from 12 to Pending Backport

This now leaks memory.. presumably stray Object's the cache that never get cleaned up.

Let's just increase the object count to be something that's pretty big? Or auto-scale based on the cache size?

#9 Updated by Ian Colle over 9 years ago

  • Status changed from Pending Backport to 12

#10 Updated by Sage Weil over 9 years ago

  • Status changed from 12 to In Progress

#11 Updated by Josh Durgin over 9 years ago

  • Backport deleted (emperor, dumpling)

#12 Updated by Sage Weil over 9 years ago

  • Priority changed from Urgent to High

#13 Updated by Sage Weil about 9 years ago

  • Project changed from Ceph to rbd
  • Category deleted (librbd)
  • Source changed from other to Development

#14 Updated by Josh Durgin almost 9 years ago

  • Status changed from In Progress to 12
  • Assignee deleted (Josh Durgin)

#15 Updated by Jason Dillaman over 8 years ago

  • Status changed from 12 to In Progress
  • Assignee set to Jason Dillaman

#16 Updated by Jason Dillaman over 8 years ago

  • Backport set to hammer,firefly

#17 Updated by Josh Durgin over 8 years ago

  • Status changed from In Progress to Pending Backport

commit:0b378942c4f1b79cb65967f2d3466728ca1c8d5b

#20 Updated by Nathan Cutler about 8 years ago

  • Status changed from Pending Backport to Resolved
  • Regression set to No

Also available in: Atom PDF