Ceph » rbd

FIO workload:

[root@sm-rhel6-template ceph-perf]# cat fio-small-write.sh
#!/bin/bash

mkdir -p /root/ceph-perf

1. 4KB
   fio --rw=write -ioengine=libaio --runtime=300 --numjobs=1 --direct=1 --bs=4K --iodepth=16 --size 8G --filename=/dev/vdb --name write-4KB > /root/ceph-perf/write-4KB.txt

##--- --- ---##

1. Output with raw RBD device (/dev/vdb)
   [root@sm-rhel6-template ceph-perf]# cat write-4KB.txt_vdb
   write-4KB: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
   fio-2.1.2
   Starting 1 process

write-4KB: (groupid=0, jobs=1): err= 0: pid=27372: Thu Oct 24 10:30:16 2013
write: io=1971.6MB, bw=6729.4KB/s, iops=1682, runt=300011msec
slat (usec): min=3, max=4873, avg=27.21, stdev=22.75
clat (msec): min=2, max=361, avg= 9.48, stdev= 5.43
lat (msec): min=2, max=361, avg= 9.51, stdev= 5.43
clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 15], 95.00th=[ 16], | 99.00th=[ 20], 99.50th=[ 28], 99.90th=[ 63], 99.95th=[ 83], | 99.99th=[ 233]
bw (KB /s): min= 2231, max= 9473, per=100.00%, avg=6740.76, stdev=818.43
lat (msec) : 4=0.29%, 10=73.82%, 20=24.92%, 50=0.79%, 100=0.15%
lat (msec) : 250=0.02%, 500=0.01%
cpu : usr=1.71%, sys=6.52%, ctx=201313, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=504718/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
WRITE: io=1971.6MB, aggrb=6729KB/s, minb=6729KB/s, maxb=6729KB/s, mint=300011msec, maxt=300011msec

Disk stats (read/write):
vdb: ios=87/504637, merge=0/0, ticks=186/4740784, in_queue=4740508, util=100.00%

##--- --- ---##

1. Output with XFS on RBD device (mkfs.xfs -f /dev/vdb; mount -t xfs /dev/vdb /mnt)
   [root@sm-rhel6-template ceph-perf]# cat write-4KB.txt_vdb-xfs1
   write-4KB: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
   fio-2.1.2
   Starting 1 process
   write-4KB: Laying out IO file(s) (1 file(s) / 8192MB)

write-4KB: (groupid=0, jobs=1): err= 0: pid=2166: Thu Oct 24 11:52:14 2013
write: io=8192.0MB, bw=77276KB/s, iops=19318, runt=108554msec
slat (usec): min=3, max=82556, avg=15.98, stdev=425.89
clat (usec): min=115, max=112785, avg=806.26, stdev=4934.66
lat (usec): min=140, max=112791, avg=823.62, stdev=4964.81
clat percentiles (usec): | 1.00th=[ 153], 5.00th=[ 197], 10.00th=[ 231], 20.00th=[ 253], | 30.00th=[ 266], 40.00th=[ 274], 50.00th=[ 286], 60.00th=[ 306], | 70.00th=[ 358], 80.00th=[ 474], 90.00th=[ 636], 95.00th=[ 732], | 99.00th=[ 7904], 99.50th=[43264], 99.90th=[73216], 99.95th=[76288], | 99.99th=[87552]
bw (KB /s): min= 1396, max=136936, per=100.00%, avg=78057.80, stdev=36881.91
lat (usec) : 250=18.32%, 500=63.44%, 750=13.91%, 1000=2.59%
lat (msec) : 2=0.45%, 4=0.08%, 10=0.24%, 20=0.18%, 50=0.37%
lat (msec) : 100=0.42%, 250=0.01%
cpu : usr=4.92%, sys=22.10%, ctx=547445, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=2097152/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
WRITE: io=8192.0MB, aggrb=77275KB/s, minb=77275KB/s, maxb=77275KB/s, mint=108554msec, maxt=108554msec

Disk stats (read/write):
vdb: ios=0/2095270, merge=0/38, ticks=0/1018999, in_queue=1017834, util=77.04%

With multiple runs of same test on RBD device with XFS, we get following outputs..

fio --rw=write -ioengine=libaio --runtime=300 --numjobs=1 --direct=1 --bs=4K --iodepth=16 --size 8G --filename=/dev/vdb --name write-4KB > /root/ceph-perf/write-4KB.txt

1. NEW IOPS for Sequential 4KB Write with rbd cache writethrough until flush = true
   Sequential Write 1 VM 19318
   Sequential Write 1 VM 27151
   Sequential Write 1 VM 26644
   Sequential Write 1 VM 23256