Project

General

Profile

Bug #15912

An OSD was seen getting ENOSPC even with osd_failsafe_full_ratio passed

Added by David Zafman over 1 year ago. Updated 2 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
05/17/2016
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
kraken, jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

The value of osd_failsafe_full_ratio only restricts new client ops by default after 97% full condition. Could it be that an OSD with large journal could have enough pending filestore data updates that the 3% isn't enough to absorb those updates?

Should new client operations be restricted based on journal size? We could make the value of osd_failsafe_full_ratio an over-ride with default to 0 (use computed value).


Related issues

Related to RADOS - Bug #18687: bluestore: ENOSPC writing to XFS block file on smithi Resolved 01/26/2017
Related to Ceph - Bug #16878: filestore: utilization ratio calculation does not take journal size into account Resolved 08/01/2016
Related to Ceph - Feature #15910: Increase the default value of mon_osd_min_in_ratio Resolved 05/17/2016
Related to Ceph - Bug #19682: Additional full fixes Resolved 04/18/2017
Related to Ceph - Bug #19733: clean up min/max span warning Resolved 04/20/2017
Copied to Ceph - Backport #19265: jewel: An OSD was seen getting ENOSPC even with osd_failsafe_full_ratio passed Resolved
Copied to Ceph - Backport #19340: kraken: An OSD was seen getting ENOSPC even with osd_failsafe_full_ratio passed Resolved

History

#1 Updated by David Zafman over 1 year ago

  • Assignee set to David Zafman

#2 Updated by David Zafman over 1 year ago

Potentially backfill or recovery used the remaining space.

#3 Updated by Ian Colle 8 months ago

  • Priority changed from Normal to Urgent

#4 Updated by David Zafman 8 months ago

https://github.com/ceph/ceph/pull/13425

There are multiple issues to address. This pull requests addresses some of them.

#5 Updated by Nathan Cutler 8 months ago

  • Related to Bug #16878: filestore: utilization ratio calculation does not take journal size into account added

#6 Updated by David Zafman 8 months ago

  • Related to Bug #18687: bluestore: ENOSPC writing to XFS block file on smithi added

#7 Updated by David Zafman 8 months ago

  • Related to deleted (Bug #16878: filestore: utilization ratio calculation does not take journal size into account)

#8 Updated by David Zafman 8 months ago

  • Related to Bug #16878: filestore: utilization ratio calculation does not take journal size into account added

#9 Updated by David Zafman 8 months ago

  • Status changed from New to Resolved

#10 Updated by David Zafman 8 months ago

  • Status changed from Resolved to Pending Backport
  • Backport set to kraken, jewel

#11 Updated by David Zafman 8 months ago

  • Related to Feature #15910: Increase the default value of mon_osd_min_in_ratio added

#12 Updated by Alexey Sheplyakov 7 months ago

  • Copied to Backport #19265: jewel: An OSD was seen getting ENOSPC even with osd_failsafe_full_ratio passed added

#13 Updated by Nathan Cutler 7 months ago

  • Copied to Backport #19340: kraken: An OSD was seen getting ENOSPC even with osd_failsafe_full_ratio passed added

#14 Updated by David Zafman 6 months ago

  • Related to Bug #19682: Additional full fixes added

#15 Updated by David Zafman 6 months ago

  • Related to Bug #19698: cephtool/test.sh error on full tests added

#16 Updated by David Zafman 6 months ago

We should backport the 2 trackers/pulls in this order:

bug #15912 Pull https://github.com/ceph/ceph/pull/13425

bug #19733 Pull https://github.com/ceph/ceph/pull/14611

#17 Updated by David Zafman 6 months ago

  • Related to deleted (Bug #19698: cephtool/test.sh error on full tests)

#18 Updated by David Zafman 6 months ago

  • Related to Bug #19733: clean up min/max span warning added

#19 Updated by Nathan Cutler 2 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF