Project

General

Profile

Bug #44937

bluestore rocksdb max_background_compactions regression in 12.2.13

Added by Dan Hill almost 4 years ago. Updated almost 4 years ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
Yes
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

In pr#29027 [0] Mark called out a rocksdb pr#3926 [1] where concurrent compactions can result in output files that have overlapping key ranges.

The rocksdb concurrency issue is fixed in Mimic onwards, but not Luminous.

In the 12.2.13 point release, pr#30149 [2] sets 'max_background_compactions=2'. This should be reverted back to 1.

[0] https://github.com/ceph/ceph/pull/29027#issue-297158998
[1] https://github.com/facebook/rocksdb/pull/3926
[2] https://github.com/ceph/ceph/pull/30149

History

#1 Updated by Dan Hill almost 4 years ago

I know Luminous is EOL. Opened this issue to assess the regression for severity.

The rocksdb issue is an edge case but seems dangerous.

#2 Updated by Greg Farnum almost 4 years ago

  • Project changed from Ceph to bluestore

#3 Updated by Igor Fedotov almost 4 years ago

Looks like rocksdb fix [2] isn't present in mimic as well. Hence setting new default for max_background_compactions needs a reversion in mimic too.

#4 Updated by Josh Durgin almost 4 years ago

Users have been using >=2 (up to 8) background_compactions in rocksdb long before we set the default to 2, without any problems.

Although there is a bug in rocksdb that Mark pointed out, it does not seem to be a very likely one to hit in practice. Have you seen many users actually run into the problem?

#5 Updated by Neha Ojha almost 4 years ago

  • Status changed from New to Need More Info

#6 Updated by Dan Hill almost 4 years ago

  • Target version deleted (v12.2.14)
  • Affected Versions v13.2.7 added

@Igor Yeah, you are correct. The pr#3926 commits landed after Mimic:

Luminous: v4.9-960-ge15382c09
Mimic:    v5.8-561-g5ec382b91
pr#3926:  v5.8-709-g4420df4b0
Nautilus: v6.1.2

@Josh No, I have not seen this in the wild.

I'd argue that defaults should generally be conservative and should not expose users to known issues.

That said, it sounds like the option has proven to be safe. It would be interesting to know how much run-time/exposure max_background_compactions > 1 has actually seen on Luminous. Hopefully the new telemetry data can help answer questions like this in the future.

An alternative to reverting max_background_compactions would be to cherry-pick the missing rocksdb commits. At a glance, it looks like they would cleanly apply.

Also available in: Atom PDF