Project

General

Profile

Bug #24361

auto compaction on rocksdb should kick in more often

Added by Kefu Chai 8 months ago. Updated 8 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
Performance/Resource Usage
Target version:
-
Start date:
05/31/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
luminous, mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:

Description

in rocksdb, by default, "max_bytes_for_level_base" is 256MB, "max_bytes_for_level_multiplier" is 10. so with this setting, the limit of each level of a rocksdb would look like

  1. L0: in memory
  2. L1: 256MB
  3. L2: 2.56 GB
  4. L3: 25.6 GB
  5. L4: 256 GB
  6. L5: 2.56 TB
  7. L6: 25.6 TB

for monitor, 2.56 GB is relative large even for a large cluster. depending on the application of OSD, i'd say 2.56 GB is quite large for omap even taking the load of rgw into consideration.

in the case of monitor, if the cluster has been running for a long time in a large scale deployment, there is chance that the old and stale data could be migrated to L3. and new K/V data come in, the are written to lower level, like L0, L1. like

  1. L1: 250MB
  2. L2: 2 GB
  3. L3: 25 GB
  4. L4: 25 GB // stale data. non-user data

then we will be suffering from "space amplification". as the space amplification is (25 + 2 + 0.25 + 25) / (25 + 2 + 0.25) = 1.91 .

and the auto compaction does not help in this case, as none of sizes exceeds max_bytes limit. so a more flexible approach is to enable the dynamic level size for compaction.

----
[0] https://rocksdb.org/blog/2015/07/23/dynamic-level.html
[1] https://github.com/facebook/rocksdb/wiki/Leveled-Compaction


Related issues

Copied to RADOS - Backport #24374: luminous: mon: auto compaction on rocksdb should kick in more often Resolved
Copied to RADOS - Backport #24375: mimic: mon: auto compaction on rocksdb should kick in more often Resolved

History

#1 Updated by Kefu Chai 8 months ago

  • Description updated (diff)

#2 Updated by Kefu Chai 8 months ago

  • Status changed from New to Need Review

#3 Updated by Kefu Chai 8 months ago

  • Status changed from Need Review to Pending Backport

#4 Updated by Kefu Chai 8 months ago

  • Copied to Backport #24374: luminous: mon: auto compaction on rocksdb should kick in more often added

#5 Updated by Kefu Chai 8 months ago

  • Copied to Backport #24375: mimic: mon: auto compaction on rocksdb should kick in more often added

Also available in: Atom PDF