"files": 30TB of data constituted of dirs containing 3.5GB dispatched into 87 files ( 2 of 920M, 6 > 100M, 4 > 10M, 7 > 1M, 68 < 1M ) The factor is the delta of "total raw" on the delta of "pool used". For the given pool, the theorical factor is 24/21 = 1.143x 3 different object size and stripe are tested with setfattr (stripe_count=1): obj_size 672MB, obj_stripe 1344kB (21*32M, 21*64k): -------------------------------------------------- rsync 900GB of files: ~88MB/s 2.45x => rem: disfunctionning count (du=123G, ceph df=52G pool used, 59G pool raw, 117G all raw) => see ticket #24402 obj_size 84MB, obj_stripe 1344kB (21*4M, 21*64k): -------------------------------------------------- rsync 5TB of files: 2.57x dd 10000x1kB urand: 1536x (like in theory: 1344kB*24/21 = 1.5Mo = minimum size fixed by stripe) => ok ! dd 10000x129kB urand: 1408x (like above) => ok ! dd 10000x1344kB urand: 1.25x (expecting 1.14x) => ok if rounding effect dd 10000x2MB urand: 1.54x (like above with 2 stripes) => ok ! dd 10000x4MB urand: 1.54x (like above with 4 stripes) => ok ! dd 10000x5376kB urand: 1.15x (expecting 1.14x) => ok ! dd 1000x84MB urand: 1.098x (expecting 1.14x) => ??????? (repeated with similar result: 1.061 and 1.110) dd 1000x(84MB+4kB) urand: 1.146 (64 full stripes+0.3% of stripe => expecting 1.16) => ok if rounding effects dd 1000x(84MB+132kB) urand: 1.157 (expecting 1.159) => ok ! dd 1000x126MB urand: 1.21x (96 full stripes, expecting 1.14x) => 5% increase acceptable dd 100x920MB file bs=128k: ~1.8MB/s 4.51x dd 100x920MB file bs=1M: ~11MB/s 2.24x cp 100x920MB file (bs=1M): ~80MB/s 2.20x dd 100x920MB file bs=4M: ~16MB/s 1.42x dd 100x920MB file bs=84M: ~24MB/s 1.31x obj_size 4MB, obj_stripe 4MB: ----------------------------- cp 400GB of files: 1.35x dd 100x920MB file bs=128k: ~1.7MB/s 4.95x dd 100x920MB file bs=1M: ~9MB/s 2.19x dd 100x920MB file bs=4M: ~40MB/s 1.48x