Ceph

I've done some fio testing to motivate this further. The fio config below should simulate the case where the journal is co-located on a spinning disk with the filestore.

[global]
  ioengine=libaio
  invalidate=1
  runtime=20
#
  [deepscrub]
  direct=0
  iodepth=8
  bs=512k
  rw=read
  size=4g
  prioclass=2 # to be varied
  prio=4 # to be varied
#
  [writeobj]
  direct=1
  sync=1 # not sure if an OSD journal is O_SYNC?
  bs=4k
  rw=write
  rate_iops=10
  size=1g
  numjobs=2
  prioclass=2 # to be varied
  prio=4 # to be varied

This configures one scrubbing thread who reads in 512k chunks, and two direct IO writers who write at 10 IOPS each. The goal is then to optimise the latency of those 10 IOPS in the write threads.

Using cfq (from RHEL6.5) and default io priorities (be/4), I get:

lat (usec): min=205 , max=304311 , avg=51303.22, stdev=46714.35

Using cfq with scrub prio=be/7 and writer prio be/1, I get:

lat (usec): min=245 , max=287430 , avg=52490.84, stdev=47354.12

Using cfq with scrub prio=idle and writer prio=be/4, I get:

lat (usec): min=273 , max=225333 , avg=6272.16, stdev=29405.21

and finally with cfq and scrub prio=be/4 and writer prio=realtime, I get:

lat (usec): min=281 , max=215453 , avg=6802.68, stdev=29901.80

We see that when the scrubber and writers are in the same prioclass (best effort), no matter the priority, the write latency is ~50ms. By either putting the scrubber into the idle class, or by putting the writers into realtime, the write latency improves to 6-7ms.

I would therefore propose to make the prioclass and prio configurable for each of the op and disk thread pools.

Hi,

The backport to dumpling is missing the commit which provides the new configurable: https://github.com/ceph/ceph/commit/987ad133415aa988061c95259f9412b05ce8ac7e

And the backport to firefly is missing completely.

Cheers, Dan