Bug #7385
closed
Objectcacher setting max object counts too low
Added by Mark Nelson about 10 years ago.
Updated almost 9 years ago.
Description
It appears that the objectcacher is setting max object counts based on the max dirty data size and object size. With default RBD block size and RBD cache values, this was max objects = 42 according to objectcacher logs. In performance testing performed on our new test hardware, this caused severe rbd cache thrashing with high amounts of concurrent IO (8 threads with io_depth = 16 each). With lower amounts of concurrent IO, journal writes on the OSDs were on average around 400-450K in size (with a default 512 max_sectors_kb value). With higher concurrent IOs, this decreased to around 12-16K default journal write size and significantly lower performance.
This may also explain the odd graphs that we saw in our cuttlefish testing with multiple volumes on a QEMU/KVM guest as the io depth increased:
http://ceph.com/wp-content/uploads/2014/07/cuttlefish-rbd_btrfs-write-0004K.png
This can likely be worked around in current code by increasing the rbd dirty cache limits, but ultimately may be fixed by changing the behavior for how object limits are set in the objectcacher.
- Category changed from ObjectCacher to librbd
This is set by librbd or ceph-fuse's Client.cc after creating the objectcacher. There's already a config option for Client.cc, so just librbd needs fixing.
Does it make sense to be exposing this to librbd/client.cc at all vs just directly setting it via a config option? (or even eliminating it?)
- Priority changed from Normal to Urgent
- Assignee set to David Zafman
- Status changed from New to Fix Under Review
- Assignee changed from David Zafman to Josh Durgin
- Backport set to emperor, dumpling
- Status changed from Fix Under Review to Pending Backport
- Status changed from Pending Backport to 12
- Status changed from 12 to Pending Backport
This now leaks memory.. presumably stray Object's the cache that never get cleaned up.
Let's just increase the object count to be something that's pretty big? Or auto-scale based on the cache size?
- Status changed from Pending Backport to 12
- Status changed from 12 to In Progress
- Backport deleted (
emperor, dumpling)
- Priority changed from Urgent to High
- Project changed from Ceph to rbd
- Category deleted (
librbd)
- Source changed from other to Development
- Status changed from In Progress to 12
- Assignee deleted (
Josh Durgin)
- Status changed from 12 to In Progress
- Assignee set to Jason Dillaman
- Backport set to hammer,firefly
- Status changed from In Progress to Pending Backport
commit:0b378942c4f1b79cb65967f2d3466728ca1c8d5b
- Status changed from Pending Backport to Resolved
- Regression set to No
Also available in: Atom
PDF