Bug #6631
closeddisabling writethrough until flush appears to disable RBD cache
0%
Description
Recently we saw a report that when using fio to perform 4K sequential direct IO writes to files on an XFS filesystem on top of RBD, writethrough until flush had to be enabled to achieve good performance. Based on the cache hit performance counter in the RBD admin socket, it appears that without writethrough until flush set RBD cache was not being used despite the ceph.conf and xml file apparently being set correctly.
Updated by Amit Vijairania over 10 years ago
On Nova node..
// Ceph.conf
[global]
fsid = c14be55f-f608-4f76-aaa3-925b07a72e43
mon initial members = alln01-cc-ceph-001,alln01-cc-ceph-003,alln01-cc-ceph-018,alln01-cc-ceph-035,alln01-cc-ceph-037
mon host = 10.123.10.11,10.123.10.13,10.123.10.28,10.123.10.45,10.123.10.47
public network = 10.123.8.0/24
cluster network = 10.123.9.0/24
[client]
rbd cache = true
rbd cache writethrough until flush = true
rbd cache size = 33554432
rbd cache max dirty = 25165824
rbd cache target dirty = 16777216
rbd cache max dirty age = 1
admin socket = /var/run/ceph/rbd-$pid.asok
[root@sm-rhel6-template ceph-perf]# cat fio-small-write.sh
fio --rw=write -ioengine=libaio --runtime=300 --numjobs=1 --direct=1 --bs=4K --iodepth=16 --size 8G --filename=/mnt/fuzzy.in --name write-4KB > /root/ceph-perf/write-4KB.txt
// IOPS for Sequential 4KB Write with "rbd cache writethrough until flush = true"
write-4KB: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
fio-2.1.2
Starting 1 process
write-4KB: Laying out IO file(s) (1 file(s) / 8192MB)
write-4KB: (groupid=0, jobs=1): err= 0: pid=2780: Thu Oct 24 12:08:48 2013
write: io=8192.0MB, bw=106577KB/s, iops=26644, runt= 78709msec
slat (usec): min=3, max=5098, avg=12.05, stdev=17.97
clat (usec): min=115, max=82067, avg=585.98, stdev=3097.62
lat (usec): min=140, max=82073, avg=598.61, stdev=3097.45
clat percentiles (usec):
| 1.00th=[ 151], 5.00th=[ 191], 10.00th=[ 225], 20.00th=[ 249],
| 30.00th=[ 258], 40.00th=[ 270], 50.00th=[ 278], 60.00th=[ 294],
| 70.00th=[ 318], 80.00th=[ 422], 90.00th=[ 612], 95.00th=[ 692],
| 99.00th=[ 1320], 99.50th=[22656], 99.90th=[50944], 99.95th=[54528],
| 99.99th=[67072]
bw (KB /s): min=85560, max=141307, per=100.00%, avg=106717.11, stdev=7822.90
lat (usec) : 250=21.06%, 500=63.52%, 750=12.39%, 1000=1.74%
lat (msec) : 2=0.34%, 4=0.07%, 10=0.15%, 20=0.18%, 50=0.41%
lat (msec) : 100=0.13%
cpu : usr=5.86%, sys=28.28%, ctx=413723, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=2097152/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=8192.0MB, aggrb=106577KB/s, minb=106577KB/s, maxb=106577KB/s, mint=78709msec, maxt=78709msec
Disk stats (read/write):
vdb: ios=0/2097107, merge=0/35, ticks=0/898360, in_queue=897853, util=92.57%
// IOPS for Sequential 4KB Write without "rbd cache writethrough until flush = true"
write-4KB: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
fio-2.1.2
Starting 1 process
write-4KB: Laying out IO file(s) (1 file(s) / 8192MB)
write-4KB: (groupid=0, jobs=1): err= 0: pid=2027: Thu Oct 24 12:23:48 2013
write: io=2501.2MB, bw=8535.7KB/s, iops=2133, runt=300059msec
slat (usec): min=3, max=199003, avg=144.89, stdev=2849.77
clat (usec): min=108, max=287391, avg=7201.73, stdev=20748.75
lat (usec): min=135, max=287566, avg=7381.62, stdev=21013.91
clat percentiles (usec):
| 1.00th=[ 149], 5.00th=[ 209], 10.00th=[ 237], 20.00th=[ 247],
| 30.00th=[ 258], 40.00th=[ 266], 50.00th=[ 278], 60.00th=[ 314],
| 70.00th=[ 572], 80.00th=[ 668], 90.00th=[ 6560], 95.00th=[71168],
| 99.00th=[77312], 99.50th=[81408], 99.90th=[93696], 99.95th=[109056],
| 99.99th=[160768]
bw (KB /s): min= 453, max=120494, per=100.00%, avg=8905.41, stdev=26238.10
lat (usec) : 250=22.87%, 500=44.58%, 750=14.93%, 1000=0.86%
lat (msec) : 2=0.24%, 4=1.62%, 10=5.74%, 20=0.10%, 50=0.26%
lat (msec) : 100=8.71%, 250=0.08%, 500=0.01%
cpu : usr=1.14%, sys=4.62%, ctx=1493701, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=640300/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=2501.2MB, aggrb=8535KB/s, minb=8535KB/s, maxb=8535KB/s, mint=300059msec, maxt=300059msec
Disk stats (read/write):
vdb: ios=0/640241, merge=0/41, ticks=0/278946, in_queue=278590, util=13.73%
Updated by Amit Vijairania over 10 years ago
More repetition of tests..
// IOPS for Sequential 4KB Write with "rbd cache writethrough until flush = true"
Sequential Write 1 VM 19318
Sequential Write 1 VM 27151
Sequential Write 1 VM 26644
Sequential Write 1 VM 23256
// IOPS for Sequential 4KB Write without "rbd cache writethrough until flush = true"
Sequential Write 1 VM 2133
Sequential Write 1 VM 2256
Updated by Sage Weil over 9 years ago
- Status changed from New to Need More Info
- Source changed from other to Community (user)
Amit Vijairania wrote:
More repetition of tests..
// IOPS for Sequential 4KB Write with "rbd cache writethrough until flush = true"
Sequential Write 1 VM 19318
Sequential Write 1 VM 27151
Sequential Write 1 VM 26644
Sequential Write 1 VM 23256// IOPS for Sequential 4KB Write without "rbd cache writethrough until flush = true"
Sequential Write 1 VM 2133
Sequential Write 1 VM 2256
any chance you can repeat this test with a recent version? we fixed some efficiently problems with writeback that may have been responsible for this.
Updated by Sage Weil over 9 years ago
- Status changed from Need More Info to Closed
pretty sure this is resolved now. if not, let us know so we can reopen!