Bug #36268
closed
Unable to recover from ENOSPC in BlueFS
Added by Igor Fedotov over 5 years ago.
Updated almost 4 years ago.
Description
Under heavy load and full DB volume BlueStore might fall into the state where it lacks additional space for BlueFS even if the space is still available at block device.
This is cased by the "lazy" behavior of free space rebalancing - it happens periodically in background rather than on demand.
On the first allocation failure OSD asserts and then is unable to restart since log replay during BlueFS open needs the space as well but rebalance is still not executed.
Then assertion again and hence getting a sort of unrecoverable deadlock for OSD.
- Status changed from New to Pending Backport
- Backport set to mimic,luminous
- Copied to Backport #36640: luminous: Unable to recover from ENOSPC in BlueFS added
- Copied to Backport #36641: mimic: Unable to recover from ENOSPC in BlueFS added
- Status changed from Pending Backport to In Progress
In fact previously mentioned PR is just a workaround to be able to manually fix the issue.
Working on the actual solution to fix BlueFS allocation strategy.
- Status changed from In Progress to Fix Under Review
- Status changed from Fix Under Review to Resolved
- Status changed from Resolved to Pending Backport
Sage, did you mean to cancel the mimic and luminous backports when you changed the status to Resolved?
- Status changed from Pending Backport to Resolved
- Backport deleted (
mimic,luminous)
Sage Weil wrote:
Alternative fix for mimic and luminous: https://github.com/ceph/ceph/pull/26735
hello，sage weil , i have meet the same issue before in Lumious and i have merged the new patch you mentioned, but it is unuseful. restart the osd have the same assert.Is there any other way to restore OSD such as clean up bluefs size or
expand bluefs size
- Pull request ID set to 25132
Also available in: Atom
PDF