Project

General

Profile

Bug #16878

Updated by Nathan Cutler over 7 years ago

On systems with abnormally large journals, it is possible to fill OSDs to 100%. This is because the utilization ratio calculation does not take the journal size into account.

This can be reproduced by creating a journals that are 10% of the OSD size and then bombarding the cluster with writes (e.g. <code>rados bench --no-cleanup</code>). The OSDs will fill up to 100% and then crash.

Furthermore, they cannot be restarted in this state. The documentation at http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/ describes a technique for addressing this 100% full OSD situation, involving deletion of PGs to free up space.

However, in this case, even when some space is created by deleting PGs, the OSD still refuses to start because the journal is full of queued writes. These get applied as the OSD is coming up, causing it to crash again.

The proposed solution is to take the journal size into account when calculating the utilization ratio.

Perhaps also add some journal sizing guidance to the documentation. When SSDs are available, there is a temptation to use up all their available space by allocating it to journals.

Back