Project

General

Profile

Bug #15259

performance: journaling results in 4X slowdown when writes are not blocked by cache

Added by Jason Dillaman about 8 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Using bench-write on a non-journaled image:

$ ./rbd bench-write --debug-ms "0/0" --debug-rbd "0/0" --io-size 4K --io-total 256M --io-pattern=rand foo
bench-write  io_size 4096 io_threads 16 bytes 134217728 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1      5216   4605.07  18862378.80
    2      5285   1924.76  7883825.66
    3      5300   1690.71  6925149.49
    4      5319   1280.93  5246679.94
    5      5347   1051.46  4306784.27
    6      5386     30.50  124909.44   <-- cache full

Using bench-write on a journaled image:

$ ./rbd bench-write --debug-ms "0/0" --debug-rbd "0/0" --io-size 4K --io-total 256M --io-pattern=rand foo
bench-write  io_size 4096 io_threads 16 bytes 134217728 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1      1079   1065.83  4365646.73  <-- cache not full, librbd performance issue 
    2      1120    544.04  2228372.40
    3      1141    384.60  1575301.78
    4      1161    292.71  1198927.04
    5      1186    235.39  964148.71
    6      1207     25.61  104896.51

Related issues

Copied to rbd - Backport #17484: jewel: performance: journaling results in 4X slowdown when writes are not blocked by cache Resolved

History

#1 Updated by Jason Dillaman almost 8 years ago

Running against an NVMe-backed OSD:

Using bench-write on a non-journaled image:

bench-write  io_size 4096 io_threads 32 bytes 1073741824 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1      7725   7757.85  31776173.64
    2     10693   5362.08  21963077.47
    3     13940   4652.49  19056599.23
    4     17349   4344.96  17796952.89
    5     20074   3999.79  16383145.79
    6     23014   3057.80  12524730.49
    7     25834   3028.29  12403869.46
    8     28915   2994.62  12265949.20
    9     32334   2997.19  12276473.83
   10     35267   3053.52  12507219.52
   11     38189   3034.98  12431291.38
   12     41148   3061.97  12541835.21
   13     44385   3096.19  12681975.53
   14     47255   2979.24  12202981.57
   15     50356   3018.20  12362527.53
   16     53953   3152.82  12913950.43
   17     57234   3142.69  12872474.04
   18     61199   3359.15  13759087.74
   19     64701   3490.79  14298278.70
   20     67842   3496.80  14322910.80

Using bench-write on a journaled image:

bench-write  io_size 4096 io_threads 32 bytes 1073741824 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1      3683   3714.73  15215530.79
    2      5969   2999.75  12286989.65
    3      7867   2632.91  10784410.09
    4      9637   2416.68  9898737.65
    5     11520   2310.29  9462946.68
    6     13374   1930.61  7907759.11
    7     15094   1825.15  7475803.49
    8     16899   1806.42  7399101.39
    9     18097   1692.18  6931184.52
   10     19630   1622.07  6643991.56
   11     21360   1598.74  6548421.07
   12     22819   1544.92  6327975.09
   13     24463   1511.83  6192462.47
   14     26033   1578.54  6465696.81
   15     27237   1513.26  6198307.12
   16     28619   1454.54  5957789.98
   17     30017   1439.66  5896867.63
   18     31549   1418.10  5808524.91
   19     33118   1424.65  5835350.72
   20     34439   1441.66  5905033.71
   21     36239   1525.12  6246883.57
   22     38130   1621.78  6642827.20
   23     40026   1695.34  6944119.78
   24     42092   1794.96  7352168.24
   25     44131   1946.14  7971397.78
   26     46091   1967.74  8059858.30
   27     48055   1986.09  8135018.95
   28     49476   1889.77  7740491.18
   29     51073   1796.12  7356892.11
   30     52782   1730.58  7088475.58

On a fast (single OSD) backend, journaling currently results in a 2x slowdown.

#2 Updated by Yuan Zhou almost 8 years ago

Jason, I think we might should expect 2x slowdown in this case, since theatrically all the writes are doubled(journal objects + RBD objects)?

#3 Updated by Jason Dillaman almost 8 years ago

@Yuan: we should only expect a 2x slowdown when the OSD is at 100% utilization. In the example above, the OSD is the bottleneck, not the backend storage.

#4 Updated by Ricardo Dias over 7 years ago

  • Assignee set to Ricardo Dias

#5 Updated by Ricardo Dias over 7 years ago

  • Status changed from New to In Progress

#6 Updated by Jason Dillaman over 7 years ago

  • Status changed from In Progress to Pending Backport

#7 Updated by Jason Dillaman over 7 years ago

#8 Updated by Jason Dillaman over 7 years ago

  • Backport set to jewel

#9 Updated by Loïc Dachary over 7 years ago

  • Copied to Backport #17484: jewel: performance: journaling results in 4X slowdown when writes are not blocked by cache added

#10 Updated by Loïc Dachary over 7 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF