Project

General

Profile

Actions

Bug #45283

closed

Kernel log flood "ceph: Failed to find inode for 1"

Added by Michael Robertson about 4 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
kceph
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Rook v1.2.7 (official chart) and ceph v14.2.9 in an AKS cluster with VMSS.
OS provided by AKS is currently Ubuntu 16.04.6 LTS, kernel 4.15.0-1071-azure.

Every block written by a pod to a ceph CSI volume generates 2 warning lines in the node's system logs (kern.log, syslog, messages, warn):
"Apr 24 09:37:46 aks-<nodename> kernel: [242123.654538] ceph: Failed to find inode for 1"
Under production load, eventually the node succumbs to DiskPressure as the drive fills up.

Jos Collin in Slack says: "It looks like the MDS is sending CEPH_MSG_CLIENT_QUOTA messages to the client with the root ino in them and the client doesn't recognise that inode (likely because it didn't mount the root). Inode 1 is typically the root of the fs."
Submitting ticket here, as requested.

One more thing: I forced OS upgrade on an experiment node to Ubuntu 18, kernel 5.0.0. The warning goes away, and writes double in speed.
However, this OS is not GA in AKS yet.

Actions

Also available in: Atom PDF