Project

General

Profile

Actions

Bug #50551

closed

Massive OSD creation: kernel parameter fs.aio-max-nr with a low value by default

Added by Juan Miguel Olmo Martínez almost 3 years ago. Updated almost 3 years ago.

Status:
Duplicate
Priority:
Normal
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

fs.aio-max-nr: The Asynchronous non-blocking I/O (AIO) feature that allows a process to initiate multiple I/O operations simultaneously without having to wait for any of them to complete. This helps boost performance for applications that are able to overlap processing and I/O.

This kernel setting is set by default to 65536 and is not enough if you want to have more than 21 OSDs running in the same host.
So use this value:

fs.aio-max-nr = max(65536, 3500 * [number of osds you want to have in the host])

In my view, does not seem sensible to let the orchestrator make this change automatically, because is affecting a kernel parameter, and therefore all the host behaviour. So i suggest to add a note in the documentation to warn about this fact.


Related issues 1 (0 open1 closed)

Related to Orchestrator - Bug #47873: /usr/lib/sysctl.d/90-ceph-osd.conf getting installed in container, rendering it ineffectiveResolvedMichael Fritch

Actions
Actions #1

Updated by Sebastian Wagner almost 3 years ago

  • Description updated (diff)
Actions #2

Updated by Dan van der Ster almost 3 years ago

We've been setting fs.aio-max-nr to 1048576 since early bluestore days with no apparent downside. That would be a simpler recommendation than your formula.

Also see https://github.com/ceph/ceph/blob/master/etc/sysctl/90-ceph-osd.conf.in

Actions #3

Updated by Sebastian Wagner almost 3 years ago

  • Related to Bug #47873: /usr/lib/sysctl.d/90-ceph-osd.conf getting installed in container, rendering it ineffective added
Actions #4

Updated by Sebastian Wagner almost 3 years ago

  • Status changed from New to Duplicate

duplicates #47873

Actions

Also available in: Atom PDF