Bug #44546
opencleanup: Can't lookup inode 1
0%
Description
I am getting these mounting a cephfs filesystem when using the 5.5 kernel (did not see them using the 3.10)
[43454.597799] ceph: Can't lookup inode 1 (err: -13)
[43824.567641] ceph: Can't lookup inode 1 (err: -13)
[43911.436290] ceph: Can't lookup inode 1 (err: -13)
[43971.797977] ceph: Can't lookup inode 1 (err: -13)
[44065.089439] ceph: Can't lookup inode 1 (err: -13)
[44146.944133] ceph: Can't lookup inode 1 (err: -13)
[44210.226530] ceph: Can't lookup inode 1 (err: -13)
[44284.735468] ceph: Can't lookup inode 1 (err: -13)
[44345.450244] ceph: Can't lookup inode 1 (err: -13)
[44415.663769] ceph: Can't lookup inode 1 (err: -13)
I heard from the mailing list it is harmless and happens when you mount a subtree of the cephfs. None the less, needs to be cleaned up/fixed
Files
Updated by Greg Farnum about 4 years ago
- Status changed from New to Need More Info
What version is the CephFS cluster this client runs against?
This sounds like the thing where clients with limited cephx caps may not be able to find a quota root outside their domain.
Updated by Paul Emmerich about 4 years ago
This seems to happen when:
- a cephfs subdirectory is mounted with the kernel
- kernel > 5.0 (not sure, may have seen it on a 4.19 setup?)
- quotas are configured
Seen it when running against Nautilus. Absolutely harmless, just a little bit spammy in the log.
Updated by Florent Flament almost 4 years ago
Hi following an upgrade to Ubuntu 20.04, I have the same issue.
- I indeed mounted a cephfs subdirectory with the kernel driver
- I am using Linux kernel 5.4 (5.4.0-28-generic)
- However, I haven't configured any quota explicitely.
Updated by Paul Emmerich almost 4 years ago
just saw this on a 5.3.13 kernel cephfs client
Updated by Luis Henriques almost 4 years ago
- File 0001-ceph-demote-quotas-warning-to-debug-message.patch 0001-ceph-demote-quotas-warning-to-debug-message.patch added
As Greg mentioned in the first comment, this seems to be a cephx configuration issue. '-13' means -EACCES (Permission denied), and it's very likely that adding 'allow r path=/' to the mds daemon config in cephx would remove this warning.
This warning is harmless unless any there are quotas set underneath the directories the user has permissions to access. For example:
- We have a directory /home/myuser
- 'myuser' has ceph.quota.max_files set to 1000
If we use the following config to mount the filesystem:
[client.myuser]
caps mds = "allow rw path=/home/myuser"
caps mon = "allow r"
caps osd = "allow rw tag cephfs data=*"
We'll get the warning but the quotas will be respected. However, if the 'home' directory also has quotas set to '500', then we'll still get the warning but the quotas of 500 won't be respected because the client can't access 'home' (it will still respect the max_files='1000').
I'm attaching a patch that demotes this warning into a debug message. I'll ping Jeff and Greg to see what they think about it.
Updated by Jeff Layton almost 4 years ago
I'm fine with demoting this log message. If we're concerned about admins not getting a hint as to the problem, then we could also consider ratelimiting it instead.
I have some questions though:
To follow up on your last example, what about the case where the client has cephx permissions for /home, but actually mounted /home/myuser? The root of the mount in that case will be the inode for /home/myuser. When the client gets a quota message from the MDS for /home, it won't have that inode in its cache. Don't we have the same problem in that case, regardless of cephx permissions?
Updated by Luis Henriques almost 4 years ago
Jeff Layton wrote:
I'm fine with demoting this log message. If we're concerned about admins not getting a hint as to the problem, then we could also consider ratelimiting it instead.
I have some questions though:
To follow up on your last example, what about the case where the client has cephx permissions for /home, but actually mounted /home/myuser? The root of the mount in that case will be the inode for /home/myuser. When the client gets a quota message from the MDS for /home, it won't have that inode in its cache. Don't we have the same problem in that case, regardless of cephx permissions?
Actually, the client will get that inode into the cache because the client will walk through the quotarealms (this happens in lookup_quotarealm_inode()). So, things will work fine in that case as long as the client has permissions to (at least) read /home.
Regarding issues with cephx admin, doc/cephfs/quota.rst already describes the problem in the 'limitations' section. I'm not sure if it really helps keeping this message ratelimited, but I can re-send the patch with that change.
Updated by Jeff Layton almost 4 years ago
Ok, in that case, let's leave it as a debug message then. I think we should probably mark your patch for stable too. I'll take it up with Ilya.
Updated by Stefan Kooman almost 4 years ago
@Luis Henriques
"As Greg mentioned in the first comment, this seems to be a cephx configuration issue. '-13' means -EACCES (Permission denied), and it's very likely that adding 'allow r path=/' to the mds daemon config in cephx would remove this warning."
^^ This might be true, but in multi tenant scenarios this would be a security violation (read access to /).
On a Ubuntu xenial system with HWE kernel (4.15.0-64-generic) the message is "ceph: Failed to find inode for 1". I have double checked that there is no quota active on a directory above the directory that is mounted (and there is a quota on the directory that is mounted).
Updated by Luis Henriques almost 4 years ago
Stefan Kooman wrote:
@Luis Henriques
"As Greg mentioned in the first comment, this seems to be a cephx configuration issue. '-13' means -EACCES (Permission denied), and it's very likely that adding 'allow r path=/' to the mds daemon config in cephx would remove this warning."
^^ This might be true, but in multi tenant scenarios this would be a security violation (read access to /).
On a Ubuntu xenial system with HWE kernel (4.15.0-64-generic) the message is "ceph: Failed to find inode for 1". I have double checked that there is no quota active on a directory above the directory that is mounted (and there is a quota on the directory that is mounted).
There's a different bug tracker for that one:
https://tracker.ceph.com/issues/45283
From my analysis, this is an issue with this Ubuntu kernel, and the bug reporter was trying to get some missing patches merged in their xenial.
Updated by Stig Telfer almost 4 years ago
I'm seeing the same message logged on CentOS 8.2 (kernel 4.18.0-193.6.3.el8_2.x86_64) mounting a submount fileshare of a Ceph Octopus CephFS using OpenStack Manila.