Project

General

Profile

Bug #36268

Unable to recover from ENOSPC in BlueFS

Added by Igor Fedotov 12 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
10/01/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

Under heavy load and full DB volume BlueStore might fall into the state where it lacks additional space for BlueFS even if the space is still available at block device.
This is cased by the "lazy" behavior of free space rebalancing - it happens periodically in background rather than on demand.
On the first allocation failure OSD asserts and then is unable to restart since log replay during BlueFS open needs the space as well but rebalance is still not executed.
Then assertion again and hence getting a sort of unrecoverable deadlock for OSD.


Related issues

Copied to bluestore - Backport #36640: luminous: Unable to recover from ENOSPC in BlueFS New
Copied to bluestore - Backport #36641: mimic: Unable to recover from ENOSPC in BlueFS New

History

#2 Updated by Sage Weil 11 months ago

  • Status changed from New to Pending Backport
  • Backport set to mimic,luminous

#3 Updated by Patrick Donnelly 11 months ago

  • Copied to Backport #36640: luminous: Unable to recover from ENOSPC in BlueFS added

#4 Updated by Patrick Donnelly 11 months ago

  • Copied to Backport #36641: mimic: Unable to recover from ENOSPC in BlueFS added

#5 Updated by Igor Fedotov 10 months ago

  • Status changed from Pending Backport to In Progress

In fact previously mentioned PR is just a workaround to be able to manually fix the issue.
Working on the actual solution to fix BlueFS allocation strategy.

#6 Updated by Igor Fedotov 10 months ago

  • Status changed from In Progress to Need Review

#7 Updated by Sage Weil 7 months ago

  • Status changed from Need Review to Resolved

#8 Updated by Nathan Cutler 7 months ago

  • Status changed from Resolved to Pending Backport

Sage, did you mean to cancel the mimic and luminous backports when you changed the status to Resolved?

#9 Updated by Sage Weil 7 months ago

  • Status changed from Pending Backport to Resolved
  • Backport deleted (mimic,luminous)

Alternative fix for mimic and luminous: https://github.com/ceph/ceph/pull/26735

#10 Updated by 鹏 张 4 months ago

Sage Weil wrote:

Alternative fix for mimic and luminous: https://github.com/ceph/ceph/pull/26735

hello,sage weil , i have meet the same issue before in Lumious and i have merged the new patch you mentioned, but it is unuseful. restart the osd have the same assert.Is there any other way to restore OSD such as clean up bluefs size or
expand bluefs size

Also available in: Atom PDF