Ceph » CephFS

"files": 30TB of data constituted of dirs containing 3.5GB dispatched into 87 files ( 2 of 920M, 6 > 100M, 4 > 10M, 7 > 1M, 68 < 1M )

The factor is the delta of "total raw" on the delta of "pool used". For the given pool, the theorical factor is 24/21 = 1.143x

3 different object size and stripe are tested with setfattr (stripe_count=1):

obj_size 672MB, obj_stripe 1344kB (21*32M, 21*64k):

--------------------------------------------------

rsync 900GB of files: ~88MB/s 2.45x => rem: disfunctionning count (du=123G, ceph df=52G pool used, 59G pool raw, 117G all raw)

=> see ticket #24402

obj_size 84MB, obj_stripe 1344kB (21*4M, 21*64k):

--------------------------------------------------

rsync 5TB of files: 2.57x

dd 10000x1kB urand: 1536x (like in theory: 1344kB*24/21 = 1.5Mo = minimum size fixed by stripe) => ok !

dd 10000x129kB urand: 1408x (like above) => ok !

dd 10000x1344kB urand: 1.25x (expecting 1.14x) => ok if rounding effect

dd 10000x2MB urand: 1.54x (like above with 2 stripes) => ok !

dd 10000x4MB urand: 1.54x (like above with 4 stripes) => ok !

dd 10000x5376kB urand: 1.15x (expecting 1.14x) => ok !

dd 1000x84MB urand: 1.098x (expecting 1.14x) => ??????? (repeated with similar result: 1.061 and 1.110)

dd 1000x(84MB+4kB) urand: 1.146 (64 full stripes+0.3% of stripe => expecting 1.16) => ok if rounding effects

dd 1000x(84MB+132kB) urand: 1.157 (expecting 1.159) => ok !

dd 1000x126MB urand: 1.21x (96 full stripes, expecting 1.14x) => 5% increase acceptable

dd 100x920MB file bs=128k: ~1.8MB/s 4.51x

dd 100x920MB file bs=1M: ~11MB/s 2.24x

cp 100x920MB file (bs=1M): ~80MB/s 2.20x

dd 100x920MB file bs=4M: ~16MB/s 1.42x

dd 100x920MB file bs=84M: ~24MB/s 1.31x

obj_size 4MB, obj_stripe 4MB:

-----------------------------

cp 400GB of files: 1.35x

dd 100x920MB file bs=128k: ~1.7MB/s 4.95x

dd 100x920MB file bs=1M: ~9MB/s 2.19x

dd 100x920MB file bs=4M: ~40MB/s 1.48x