Bug #15259
closed
performance: journaling results in 4X slowdown when writes are not blocked by cache
Added by Jason Dillaman about 8 years ago.
Updated over 7 years ago.
Description
Using bench-write on a non-journaled image:
$ ./rbd bench-write --debug-ms "0/0" --debug-rbd "0/0" --io-size 4K --io-total 256M --io-pattern=rand foo
bench-write io_size 4096 io_threads 16 bytes 134217728 pattern random
SEC OPS OPS/SEC BYTES/SEC
1 5216 4605.07 18862378.80
2 5285 1924.76 7883825.66
3 5300 1690.71 6925149.49
4 5319 1280.93 5246679.94
5 5347 1051.46 4306784.27
6 5386 30.50 124909.44 <-- cache full
Using bench-write on a journaled image:
$ ./rbd bench-write --debug-ms "0/0" --debug-rbd "0/0" --io-size 4K --io-total 256M --io-pattern=rand foo
bench-write io_size 4096 io_threads 16 bytes 134217728 pattern random
SEC OPS OPS/SEC BYTES/SEC
1 1079 1065.83 4365646.73 <-- cache not full, librbd performance issue
2 1120 544.04 2228372.40
3 1141 384.60 1575301.78
4 1161 292.71 1198927.04
5 1186 235.39 964148.71
6 1207 25.61 104896.51
Running against an NVMe-backed OSD:
Using bench-write on a non-journaled image:
bench-write io_size 4096 io_threads 32 bytes 1073741824 pattern random
SEC OPS OPS/SEC BYTES/SEC
1 7725 7757.85 31776173.64
2 10693 5362.08 21963077.47
3 13940 4652.49 19056599.23
4 17349 4344.96 17796952.89
5 20074 3999.79 16383145.79
6 23014 3057.80 12524730.49
7 25834 3028.29 12403869.46
8 28915 2994.62 12265949.20
9 32334 2997.19 12276473.83
10 35267 3053.52 12507219.52
11 38189 3034.98 12431291.38
12 41148 3061.97 12541835.21
13 44385 3096.19 12681975.53
14 47255 2979.24 12202981.57
15 50356 3018.20 12362527.53
16 53953 3152.82 12913950.43
17 57234 3142.69 12872474.04
18 61199 3359.15 13759087.74
19 64701 3490.79 14298278.70
20 67842 3496.80 14322910.80
Using bench-write on a journaled image:
bench-write io_size 4096 io_threads 32 bytes 1073741824 pattern random
SEC OPS OPS/SEC BYTES/SEC
1 3683 3714.73 15215530.79
2 5969 2999.75 12286989.65
3 7867 2632.91 10784410.09
4 9637 2416.68 9898737.65
5 11520 2310.29 9462946.68
6 13374 1930.61 7907759.11
7 15094 1825.15 7475803.49
8 16899 1806.42 7399101.39
9 18097 1692.18 6931184.52
10 19630 1622.07 6643991.56
11 21360 1598.74 6548421.07
12 22819 1544.92 6327975.09
13 24463 1511.83 6192462.47
14 26033 1578.54 6465696.81
15 27237 1513.26 6198307.12
16 28619 1454.54 5957789.98
17 30017 1439.66 5896867.63
18 31549 1418.10 5808524.91
19 33118 1424.65 5835350.72
20 34439 1441.66 5905033.71
21 36239 1525.12 6246883.57
22 38130 1621.78 6642827.20
23 40026 1695.34 6944119.78
24 42092 1794.96 7352168.24
25 44131 1946.14 7971397.78
26 46091 1967.74 8059858.30
27 48055 1986.09 8135018.95
28 49476 1889.77 7740491.18
29 51073 1796.12 7356892.11
30 52782 1730.58 7088475.58
On a fast (single OSD) backend, journaling currently results in a 2x slowdown.
Jason, I think we might should expect 2x slowdown in this case, since theatrically all the writes are doubled(journal objects + RBD objects)?
@Yuan: we should only expect a 2x slowdown when the OSD is at 100% utilization. In the example above, the OSD is the bottleneck, not the backend storage.
- Assignee set to Ricardo Dias
- Status changed from New to In Progress
- Status changed from In Progress to Pending Backport
- Copied to Backport #17484: jewel: performance: journaling results in 4X slowdown when writes are not blocked by cache added
- Status changed from Pending Backport to Resolved
Also available in: Atom
PDF