Bug #45283
closedKernel log flood "ceph: Failed to find inode for 1"
0%
Description
Rook v1.2.7 (official chart) and ceph v14.2.9 in an AKS cluster with VMSS.
OS provided by AKS is currently Ubuntu 16.04.6 LTS, kernel 4.15.0-1071-azure.
Every block written by a pod to a ceph CSI volume generates 2 warning lines in the node's system logs (kern.log, syslog, messages, warn):
"Apr 24 09:37:46 aks-<nodename> kernel: [242123.654538] ceph: Failed to find inode for 1"
Under production load, eventually the node succumbs to DiskPressure as the drive fills up.
Jos Collin in Slack says: "It looks like the MDS is sending CEPH_MSG_CLIENT_QUOTA messages to the client with the root ino in them and the client doesn't recognise that inode (likely because it didn't mount the root). Inode 1 is typically the root of the fs."
Submitting ticket here, as requested.
One more thing: I forced OS upgrade on an experiment node to Ubuntu 18, kernel 5.0.0. The warning goes away, and writes double in speed.
However, this OS is not GA in AKS yet.