Bug #36268
closedUnable to recover from ENOSPC in BlueFS
0%
Description
Under heavy load and full DB volume BlueStore might fall into the state where it lacks additional space for BlueFS even if the space is still available at block device.
This is cased by the "lazy" behavior of free space rebalancing - it happens periodically in background rather than on demand.
On the first allocation failure OSD asserts and then is unable to restart since log replay during BlueFS open needs the space as well but rebalance is still not executed.
Then assertion again and hence getting a sort of unrecoverable deadlock for OSD.
Updated by Yuri Weinstein over 5 years ago
Updated by Sage Weil over 5 years ago
- Status changed from New to Pending Backport
- Backport set to mimic,luminous
Updated by Patrick Donnelly over 5 years ago
- Copied to Backport #36640: luminous: Unable to recover from ENOSPC in BlueFS added
Updated by Patrick Donnelly over 5 years ago
- Copied to Backport #36641: mimic: Unable to recover from ENOSPC in BlueFS added
Updated by Igor Fedotov over 5 years ago
- Status changed from Pending Backport to In Progress
In fact previously mentioned PR is just a workaround to be able to manually fix the issue.
Working on the actual solution to fix BlueFS allocation strategy.
Updated by Igor Fedotov over 5 years ago
- Status changed from In Progress to Fix Under Review
Updated by Sage Weil about 5 years ago
- Status changed from Fix Under Review to Resolved
Updated by Nathan Cutler about 5 years ago
- Status changed from Resolved to Pending Backport
Sage, did you mean to cancel the mimic and luminous backports when you changed the status to Resolved?
Updated by Sage Weil about 5 years ago
- Status changed from Pending Backport to Resolved
- Backport deleted (
mimic,luminous)
Alternative fix for mimic and luminous: https://github.com/ceph/ceph/pull/26735
Updated by 鹏 张 almost 5 years ago
Sage Weil wrote:
Alternative fix for mimic and luminous: https://github.com/ceph/ceph/pull/26735
hello,sage weil , i have meet the same issue before in Lumious and i have merged the new patch you mentioned, but it is unuseful. restart the osd have the same assert.Is there any other way to restore OSD such as clean up bluefs size or
expand bluefs size