Project

General

Profile

Bug #39318

w_await high when rockdb compacting

Added by hoan nv almost 5 years ago. Updated 5 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

hi all.

I have a ceph cluster and all disk ssd.
I benchmark disk. it ok

sometime disk has w_await to 300 or 1000

This is log osd.

I see w_await high when rockdb compacting

2019-04-16 16:32:02.519 7f33aa693700  3 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/db_impl.cc:49
1]
** DB Stats **
Uptime(secs): 2289.1 total, 1334.4 interval
Cumulative writes: 225K writes, 1123K keys, 225K commit groups, 1.0 writes per commit group, ingest: 1.67 GB, 0.75 MB/s
Cumulative WAL: 225K writes, 109K syncs, 2.05 writes per sync, written: 1.67 GB, 0.75 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 118K writes, 558K keys, 118K commit groups, 1.0 writes per commit group, ingest: 979.38 MB, 0.73 MB/s
Interval WAL: 118K writes, 58K syncs, 2.03 writes per sync, written: 0.96 MB, 0.73 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent

** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      4/0   65.27 MB   1.0      0.0     0.0      0.0       0.1      0.1       0.0   1.0      0.0     70.6         2         8    0.253       0      0
  L1      4/0   214.13 MB   1.0      0.3     0.1      0.2       0.2      0.0       0.0   2.8     26.9     21.0        10         1   10.218   1598K    88K
  L2     30/0    1.45 GB   0.6      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
 Sum     38/0    1.72 GB   0.0      0.3     0.1      0.2       0.3      0.2       0.0   2.5     22.5     29.2        12         9    1.360   1598K    88K
 Int      0/0    0.00 KB   0.0      0.3     0.1      0.2       0.3      0.1       0.0   4.3     24.4     24.8        11         5    2.250   1598K    88K
Uptime(secs): 2289.1 total, 1334.4 interval
Flush(GB): cumulative 0.139, interval 0.064
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.35 GB write, 0.16 MB/s write, 0.27 GB read, 0.12 MB/s read, 12.2 seconds
Interval compaction: 0.27 GB write, 0.21 MB/s write, 0.27 GB read, 0.21 MB/s read, 11.3 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

** File Read Latency Histogram By Level [default] **

** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      4/0   65.27 MB   1.0      0.0     0.0      0.0       0.1      0.1       0.0   1.0      0.0     70.6         2         8    0.253       0      0
  L1      4/0   214.13 MB   1.0      0.3     0.1      0.2       0.2      0.0       0.0   2.8     26.9     21.0        10         1   10.218   1598K    88K
  L2     30/0    1.45 GB   0.6      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
 Sum     38/0    1.72 GB   0.0      0.3     0.1      0.2       0.3      0.2       0.0   2.5     22.5     29.2        12         9    1.360   1598K    88K
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
Uptime(secs): 2289.1 total, 0.0 interval
Flush(GB): cumulative 0.139, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.35 GB write, 0.16 MB/s write, 0.27 GB read, 0.12 MB/s read, 12.2 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

** File Read Latency Histogram By Level [default] **

Rockdb compact too slow?

Thanks

log-compact-ceph (209 KB) hoan nv, 04/22/2019 02:43 AM

Grafana - osd-write-latency.png View - disk write latency (26.7 KB) hoan nv, 04/26/2019 02:33 AM

Grafana - osd -latency.png View - osd latency (56.5 KB) hoan nv, 04/26/2019 02:33 AM

Grafana - write latency after tuning rocksdb.png View - write latency after tuning rocksdb (30.4 KB) hoan nv, 05/08/2019 03:29 AM


Related issues

Related to bluestore - Bug #23819: how to make compactions smooth Won't Fix 04/23/2018

History

#1 Updated by hoan nv almost 5 years ago

Please update issue object to 'high w_await when rockdb compact'.
Thanks

#2 Updated by Nathan Cutler almost 5 years ago

  • Subject changed from [] to w_await high when rockdb compacting

#3 Updated by hoan nv almost 5 years ago

update.

my log in attach file.

rocksdb of osd compact after 30 minutes. i thinks it too fast.

when osd has this log .

2019-04-22 08:27:00.029 7fcdf7447700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1555896420030536, "job": 1891, "event": "table_file_deletion", "file_number": 74852}

w_await ( use iostat command) increase to 500.

#4 Updated by hoan nv almost 5 years ago

#5 Updated by Sage Weil almost 5 years ago

  • Status changed from New to 4

This sounds like the SSD is just busy from compaction and/or client work. The w_await latency does sound a bit high but I don't think anything is broken. Mark, do you have any sense of what w_await values are typical during compaction?

#6 Updated by Mark Nelson almost 5 years ago

w_await is literally how long it's taking for the device to service requests. If the latency is increasing that means the device is being saturated. There's no standard for w_await times, every device will be different. FWIW I ran the log through my compaction event analyzer (https://github.com/ceph/cbt/blob/master/tools/ceph_rocksdb_log_parser.py):

$ ~/src/cbt/tools/ceph_rocksdb_log_parser.py -s SUMMARY log-compact-ceph
Compaction Statistics log-compact-ceph
Total OSD Log Duration (seconds) 3901.452
Number of Compaction Events 2
Avg Compaction Time (seconds) 16.789999
Total Compaction Time (seconds) 33.579998
Avg Output Size: (MB) 445.891469002
Total Output Size: (MB) 891.782938004
Total Input Records 17096836
Total Output Records 16772705
Avg Output Throughput (MB/s) 26.5569149041
Avg Input Records/second 509134.588671
Avg Output Records/second 499481.8232
Avg Output/Input Ratio 0.981039334397

That output MB/s rate is pretty low. I just checked my dev box with an old LSI Nytro warp SSD and see: 127MB/s and on our Incerta test nodes with Intel P3700 NVMe drives I see 175MB/s:

$ ~/src/cbt/tools/ceph_rocksdb_log_parser.py -s SUMMARY /tmp/cbt/ceph/log/osd.0.log
Compaction Statistics /tmp/cbt/ceph/log/osd.0.log
Total OSD Log Duration (seconds) 249014.234
Number of Compaction Events 903
Avg Compaction Time (seconds) 1.47857799889
Total Compaction Time (seconds) 1335.155933
Avg Output Size: (MB) 217.804265428
Total Output Size: (MB) 196677.251681
Total Input Records 1002905491
Total Output Records 722756157
Avg Output Throughput (MB/s) 175.172146374
Avg Input Records/second 913748.31391
Avg Output Records/second 623725.310976
Avg Output/Input Ratio 0.712566307625

One thing I noticed is that you seem to have bigger compaction events and smaller records (nearly as high output records/second with much lower throughput). Are you using the standard bluestore rocksdb settings?

Potentially there is good news: Adam is seeing lower write amplification during compaction when experimenting with trocksdb (https://github.com/ToshibaMemoryAmerica/trocksdb). There's no guarantee that we will be able to use it, but this may be a way in the future to reduce the amount of work rocksdb does during compaction (so instead of making the device faster, there will be less work for it to do overall).

Mark

#7 Updated by hoan nv almost 5 years ago

My config rocksdb.

bluestore_rocksdb_options = "compression=kSnappyCompression,max_write_buffer_number=16,min_write_buffer_number_to_merge=3,recycle_log_file_num=16,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=5242880,compaction_threads=40,flusher_threads=20" 

log rocksdb compacting

Uptime(secs): 32295.2 total, 985.0 interval
Cumulative writes: 4109K writes, 21M keys, 4109K commit groups, 1.0 writes per commit group, ingest: 42.37 GB, 1.34 MB/s
Cumulative WAL: 4109K writes, 1970K syncs, 2.09 writes per sync, written: 42.37 GB, 1.34 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 225K writes, 1049K keys, 225K commit groups, 1.0 writes per commit group, ingest: 2951.66 MB, 3.00 MB/s
Interval WAL: 225K writes, 109K syncs, 2.05 writes per sync, written: 2.88 MB, 3.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent

** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      4/0   47.40 MB   1.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     36.2        22        60    0.371       0      0
  L1      3/0   151.12 MB   0.8      2.9     0.7      2.2       2.3      0.1       0.0   3.1     42.3     33.5        70        14    5.011     28M  1849K
  L2     19/0    1.10 GB   0.4      1.5     0.2      1.3       1.3     -0.0       0.0   6.3     48.2     41.4        32         3   10.597     12M  4490K
 Sum     26/0    1.30 GB   0.0      4.4     0.9      3.4       4.4      0.9       0.0   5.5     36.2     36.0       124        77    1.613     41M  6339K
 Int      0/0    0.00 KB   0.0      0.2     0.1      0.1       0.2      0.1       0.0   4.2     32.2     34.3         6         5    1.157   1231K    73K
Uptime(secs): 32295.2 total, 985.0 interval
Flush(GB): cumulative 0.787, interval 0.046
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 4.37 GB write, 0.14 MB/s write, 4.39 GB read, 0.14 MB/s read, 124.2 seconds
Interval compaction: 0.19 GB write, 0.20 MB/s write, 0.18 GB read, 0.19 MB/s read, 5.8 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pendin
g_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

** File Read Latency Histogram By Level [default] **

** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      4/0   47.40 MB   1.0      0.0     0.0      0.0       0.8      0.8       0.0   1.0      0.0     36.2        22        60    0.371       0      0
  L1      3/0   151.12 MB   0.8      2.9     0.7      2.2       2.3      0.1       0.0   3.1     42.3     33.5        70        14    5.011     28M  1849K
  L2     19/0    1.10 GB   0.4      1.5     0.2      1.3       1.3     -0.0       0.0   6.3     48.2     41.4        32         3   10.597     12M  4490K
 Sum     26/0    1.30 GB   0.0      4.4     0.9      3.4       4.4      0.9       0.0   5.5     36.2     36.0       124        77    1.613     41M  6339K
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0

Uptime(secs): 32295.2 total, 0.0 interval
Flush(GB): cumulative 0.787, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 4.37 GB write, 0.14 MB/s write, 4.39 GB read, 0.14 MB/s read, 124.2 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pendin
g_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

** File Read Latency Histogram By Level [default] **

2019-04-26 09:06:57.109 7f5903d02700  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/h
uge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/compaction_job.cc:1494] [default] [JOB 79] Compacting 4@0 + 3@1 files to L1, score 1.00
2019-04-26 09:06:57.109 7f5903d02700  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/h
uge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/compaction_job.cc:1498] [default] Compaction start summary: Base version 80 Base level 0, inputs: [73347(14MB) 73343(14MB
) 73339(9813KB) 73335(9752KB)], [73329(65MB) 73330(65MB) 73331(19MB)]

2019-04-26 09:06:57.109 7f5903d02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1556244417110042, "job": 79, "event": "compaction_started", "files_L0": [73347, 73343, 73339, 73335], " 
files_L1": [73329, 73330, 73331], "score": 1, "input_data_size": 208164366}
2019-04-26 09:06:59.540 7f5903d02700  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/h
uge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 79] Generated table #73348: 495873 keys, 68983600 bytes
2019-04-26 09:06:59.541 7f5903d02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1556244419542196, "cf_name": "default", "job": 79, "event": "table_file_creation", "file_number": 73348, "file_size": 68983600, "table_properties": {"data_size": 67112684, "index_size": 1508679, "filter_size": 1396626, "raw_key_size": 44914411, "raw_average_key_size": 90, "raw_value_size": 107965452, "raw_average_value_size": 217, "num_data_blocks": 17521, "num_entries": 495873, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "278154", "kMergeOperands": "0"}}
2019-04-26 09:07:01.451 7f5903d02700  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 79] Generated table #73349: 366583 keys, 68636815 bytes
2019-04-26 09:07:01.451 7f5903d02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1556244421452332, "cf_name": "default", "job": 79, "event": "table_file_creation", "file_number": 73349, "file_size": 68636815, "table_properties": {"data_size": 67110604, "index_size": 1350785, "filter_size": 1075792, "raw_key_size": 32156342, "raw_average_key_size": 87, "raw_value_size": 113533715, "raw_average_value_size": 309, "num_data_blocks": 15659, "num_entries": 366583, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "221431", "kMergeOperands": "0"}}
2019-04-26 09:07:02.578 7f5903d02700  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/compaction_job.cc:1166] [default] [JOB 79] Generated table #73350: 534451 keys, 31215515 bytes
2019-04-26 09:07:02.578 7f5903d02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1556244422579595, "cf_name": "default", "job": 79, "event": "table_file_creation", "file_number": 73350, "file_size": 31215515, "table_properties": {"data_size": 29452481, "index_size": 1012897, "filter_size": 1406552, "raw_key_size": 27939801, "raw_average_key_size": 52, "raw_value_size": 70406148, "raw_average_value_size": 131, "num_data_blocks": 14873, "num_entries": 534451, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "271353", "kMergeOperands": "10782"}}
2019-04-26 09:07:02.578 7f5903d02700  4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/compaction_job.cc:1244] [default] [JOB 79] Compacted 4@0 + 3@1 files to L1 => 168835930 bytes
2019-04-26 09:07:02.608 7f5903d02700  4 rocksdb: (Original Log Time 2019/04/26-09:07:02.609780) [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.5/rpm/el7/BUILD/ceph-13.2.5/src/rocksdb/db/compaction_job.cc:642] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 19 0 0 0 0] max score 0.81, MB/sec: 38.1 rd, 30.9 wr, level 1, files in(4, 3) out(3) MB in(47.4, 151.1) out(161.0), read-write-amplify(7.6) write-amplify(3.4) OK, records in: 1473379, records dropped: 75769
2019-04-26 09:07:02.608 7f5903d02700  4 rocksdb: (Original Log Time 2019/04/26-09:07:02.609803) EVENT_LOG_v1 {"time_micros": 1556244422609794, "job": 79, "event": "compaction_finished", "compaction_time_micros": 5469731, "output_level": 1, "num_output_files": 3, "total_output_size": 168835930, "num_input_records": 1472676, "num_output_records": 1396907, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 19, 0, 0, 0, 0]}
2019-04-26 09:07:02.608 7f5903d02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1556244422609968, "job": 79, "event": "table_file_deletion", "file_number": 73347}
2019-04-26 09:07:02.608 7f5903d02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1556244422610039, "job": 79, "event": "table_file_deletion", "file_number": 73343}
2019-04-26 09:07:02.608 7f5903d02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1556244422610109, "job": 79, "event": "table_file_deletion", "file_number": 73339}
2019-04-26 09:07:02.608 7f5903d02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1556244422610172, "job": 79, "event": "table_file_deletion", "file_number": 73335}
2019-04-26 09:07:02.609 7f5903d02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1556244422610227, "job": 79, "event": "table_file_deletion", "file_number": 73331}
2019-04-26 09:07:02.609 7f5903d02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1556244422610283, "job": 79, "event": "table_file_deletion", "file_number": 73330}
2019-04-26 09:07:02.609 7f5903d02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1556244422610340, "job": 79, "event": "table_file_deletion", "file_number": 73329}

2 image show osd latency and disk w_await in attach file

if bluestore_rocksdb_options has default value, osd latency and disk write latency increase.

thanks.

#8 Updated by hoan nv almost 5 years ago

@Mark

Do you need more information?

#9 Updated by hoan nv almost 5 years ago

use rocksdb tuning options in tuning guide : https://ceph.com/community/bluestore-default-vs-tuned-performance-comparison/

bluestore_rocksdb_options = compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,compaction_threads=32,max_bytes_for_level_multiplier=8,flusher_threads=8,compaction_readahead_size=2MB

latency reduce to < 20 ms

image in attach file.

#10 Updated by hoan nv almost 5 years ago

i found an issue same my issue

https://tracker.ceph.com/issues/23819

#11 Updated by Neha Ojha over 4 years ago

  • Related to Bug #23819: how to make compactions smooth added

#12 Updated by Patrick Donnelly over 4 years ago

  • Status changed from 4 to New

#13 Updated by Francois Scheurer about 2 years ago

hoan nv wrote:

use rocksdb tuning options in tuning guide : https://ceph.com/community/bluestore-default-vs-tuned-performance-comparison/

[...]

latency reduce to < 20 ms

image in attach file.

Huge Thanks to Hoan for these tuning options !!!
We had also high latencies with EC8+2 pools (on nvme with nautilus 14.2.12).
These options fixed them. (at least for the time being, we did the change only 12 hours ago)

#14 Updated by Igor Fedotov 5 months ago

  • Status changed from New to Closed

Recently we had a bunch of Rocksdb tunings which hopefully fix the compaction behavior:

-https://github.com/ceph/ceph/pull/47221
-https://github.com/ceph/ceph/pull/51821
-https://github.com/ceph/ceph/pull/53343

Hence closing.

Also available in: Atom PDF