Project

General

Profile

Bug #44546

cleanup: Can't lookup inode 1

Added by none none 10 months ago. Updated 6 months ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Category:
Correctness/Safety
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature:

Description

I am getting these mounting a cephfs filesystem when using the 5.5 kernel (did not see them using the 3.10)

[43454.597799] ceph: Can't lookup inode 1 (err: -13)
[43824.567641] ceph: Can't lookup inode 1 (err: -13)
[43911.436290] ceph: Can't lookup inode 1 (err: -13)
[43971.797977] ceph: Can't lookup inode 1 (err: -13)
[44065.089439] ceph: Can't lookup inode 1 (err: -13)
[44146.944133] ceph: Can't lookup inode 1 (err: -13)
[44210.226530] ceph: Can't lookup inode 1 (err: -13)
[44284.735468] ceph: Can't lookup inode 1 (err: -13)
[44345.450244] ceph: Can't lookup inode 1 (err: -13)
[44415.663769] ceph: Can't lookup inode 1 (err: -13)

I heard from the mailing list it is harmless and happens when you mount a subtree of the cephfs. None the less, needs to be cleaned up/fixed

0001-ceph-demote-quotas-warning-to-debug-message.patch View (1.05 KB) Luis Henriques, 05/05/2020 11:32 AM

History

#1 Updated by Greg Farnum 10 months ago

  • Status changed from New to Need More Info

What version is the CephFS cluster this client runs against?

This sounds like the thing where clients with limited cephx caps may not be able to find a quota root outside their domain.

#2 Updated by Paul Emmerich 10 months ago

This seems to happen when:

  • a cephfs subdirectory is mounted with the kernel
  • kernel > 5.0 (not sure, may have seen it on a 4.19 setup?)
  • quotas are configured

Seen it when running against Nautilus. Absolutely harmless, just a little bit spammy in the log.

#3 Updated by Florent Flament 9 months ago

Hi following an upgrade to Ubuntu 20.04, I have the same issue.

  • I indeed mounted a cephfs subdirectory with the kernel driver
  • I am using Linux kernel 5.4 (5.4.0-28-generic)
  • However, I haven't configured any quota explicitely.

#4 Updated by Paul Emmerich 9 months ago

just saw this on a 5.3.13 kernel cephfs client

#5 Updated by Luis Henriques 9 months ago

As Greg mentioned in the first comment, this seems to be a cephx configuration issue. '-13' means -EACCES (Permission denied), and it's very likely that adding 'allow r path=/' to the mds daemon config in cephx would remove this warning.

This warning is harmless unless any there are quotas set underneath the directories the user has permissions to access. For example:

  • We have a directory /home/myuser
  • 'myuser' has ceph.quota.max_files set to 1000

If we use the following config to mount the filesystem:

[client.myuser]
        caps mds = "allow rw path=/home/myuser" 
        caps mon = "allow r" 
        caps osd = "allow rw tag cephfs data=*" 

We'll get the warning but the quotas will be respected. However, if the 'home' directory also has quotas set to '500', then we'll still get the warning but the quotas of 500 won't be respected because the client can't access 'home' (it will still respect the max_files='1000').

I'm attaching a patch that demotes this warning into a debug message. I'll ping Jeff and Greg to see what they think about it.

#6 Updated by Jeff Layton 9 months ago

I'm fine with demoting this log message. If we're concerned about admins not getting a hint as to the problem, then we could also consider ratelimiting it instead.

I have some questions though:

To follow up on your last example, what about the case where the client has cephx permissions for /home, but actually mounted /home/myuser? The root of the mount in that case will be the inode for /home/myuser. When the client gets a quota message from the MDS for /home, it won't have that inode in its cache. Don't we have the same problem in that case, regardless of cephx permissions?

#7 Updated by Luis Henriques 9 months ago

Jeff Layton wrote:

I'm fine with demoting this log message. If we're concerned about admins not getting a hint as to the problem, then we could also consider ratelimiting it instead.

I have some questions though:

To follow up on your last example, what about the case where the client has cephx permissions for /home, but actually mounted /home/myuser? The root of the mount in that case will be the inode for /home/myuser. When the client gets a quota message from the MDS for /home, it won't have that inode in its cache. Don't we have the same problem in that case, regardless of cephx permissions?

Actually, the client will get that inode into the cache because the client will walk through the quotarealms (this happens in lookup_quotarealm_inode()). So, things will work fine in that case as long as the client has permissions to (at least) read /home.

Regarding issues with cephx admin, doc/cephfs/quota.rst already describes the problem in the 'limitations' section. I'm not sure if it really helps keeping this message ratelimited, but I can re-send the patch with that change.

#8 Updated by Jeff Layton 9 months ago

Ok, in that case, let's leave it as a debug message then. I think we should probably mark your patch for stable too. I'll take it up with Ilya.

#9 Updated by Stefan Kooman 9 months ago

@Luis Henriques

"As Greg mentioned in the first comment, this seems to be a cephx configuration issue. '-13' means -EACCES (Permission denied), and it's very likely that adding 'allow r path=/' to the mds daemon config in cephx would remove this warning."

^^ This might be true, but in multi tenant scenarios this would be a security violation (read access to /).

On a Ubuntu xenial system with HWE kernel (4.15.0-64-generic) the message is "ceph: Failed to find inode for 1". I have double checked that there is no quota active on a directory above the directory that is mounted (and there is a quota on the directory that is mounted).

#10 Updated by Luis Henriques 9 months ago

Stefan Kooman wrote:

@Luis Henriques

"As Greg mentioned in the first comment, this seems to be a cephx configuration issue. '-13' means -EACCES (Permission denied), and it's very likely that adding 'allow r path=/' to the mds daemon config in cephx would remove this warning."

^^ This might be true, but in multi tenant scenarios this would be a security violation (read access to /).

On a Ubuntu xenial system with HWE kernel (4.15.0-64-generic) the message is "ceph: Failed to find inode for 1". I have double checked that there is no quota active on a directory above the directory that is mounted (and there is a quota on the directory that is mounted).

There's a different bug tracker for that one:

https://tracker.ceph.com/issues/45283

From my analysis, this is an issue with this Ubuntu kernel, and the bug reporter was trying to get some missing patches merged in their xenial.

#11 Updated by Stig Telfer 6 months ago

I'm seeing the same message logged on CentOS 8.2 (kernel 4.18.0-193.6.3.el8_2.x86_64) mounting a submount fileshare of a Ceph Octopus CephFS using OpenStack Manila.

Also available in: Atom PDF