Bug #19103
closedcephfs: Out of space handling
0%
Description
Cephfs needs to be more careful on a cluster with almost full OSDs. There is a delay in OSDs reporting stats, a MON marking CEPH_OSDMAP_FULL based on mon_osd_full_ratio and the MDS getting the map. The current default for mon_osd_full_ratio (95%) may or may not be sufficient to prevent running an OSD filesystem out of space before the full flag is noticed.
Currently, there is no code in MDS that checks CEPH_OSDMAP_NEARFULL. Starting to do sync writes alone at nearfull won't work because OSDs never return ENOSPC to MDS. We'd also need a flag to indicate to the OSD that MDS wants to get ENOSPC (if osd_failsafe_full_ratio has been reached) which it MDS set on file data writes only.
Updated by John Spray about 7 years ago
Could the "ENOSPC on failsafe_full_ratio" behaviour be the default? It seems like any application layer that wants to do sane ENOSPC handling would want that behaviour.
Updated by John Spray about 7 years ago
- Affected Versions v0.54b, v0.55d, v0.56, v0.57a added
Also, I'm not sure we actually need to do sync writes when the cluster is near full -- we already have machinery that handles the case of throwing ENOSPC on the fsync when someone has buffered writes that cannot be flushed because of enospc.
Updated by John Spray about 7 years ago
- Affected Versions deleted (
v0.54b, v0.55d, v0.56, v0.57a)
Believe it or not I sneezed and somehow that caused me to select some affected versions...
Updated by John Spray about 7 years ago
Looking again now that I'm a few coffees into my day -- all the cephfs enospc stuff is just aimed at providing a slicker experience to userspace (i.e. giving them ENOSPC rather than blocking ops in the Objecter), not at protecting OSDs. I don't think CephFS should be paying attention to any nearfull stuff. If OSDs are not coping with write ops when they are close to full, then that is definitely a RADOS bug.
Also, using the OSDMap full flags to block ops is an Objecter thing, not specific to cephfs. If the OSDs need all the application-facing layers to apply nearfull logic independently then we've got bigger problems.
Am I missing something or can we close this?
Updated by John Spray about 7 years ago
- Status changed from New to Won't Fix
The consensus at the moment seems to be that cephfs shouldn't be doing any nearfull special behaviour.