Bug #20340
closedcephfs permission denied until second client accesses file
0%
Description
Here is a file that client A gets permission denied during stat:
[root@aijens151 ~]# stat /mnt/puppet/aijens151.cern.ch/clone/hostgroups/paas/fix_lint/.git/objects/47/0442815e1ce9e96d66f49369877aeedd24db50 stat: cannot stat ‘/mnt/puppet/aijens151.cern.ch/clone/hostgroups/paas/fix_lint/.git/objects/47/0442815e1ce9e96d66f49369877aeedd24db50’: Permission denied
On the (single active) MDS, the permission denied looks like this:
2017-06-19 14:27:53.225362 7f5d7e9a6700 20 Session check_access stray_prior_path #100017975f1/tmp_obj_rFEkRh 2017-06-19 14:27:53.225363 7f5d7e9a6700 10 MDSAuthCap is_capable inode(path /100017975f1/tmp_obj_rFEkRh owner 993:991 mode 0100444) by caller 0:0 mask 1 new 0:0 cap: MDSAuthCaps[allow rw path="/volumes/_nogroup/4722edf2-a761-4189-b925-dbf2d44c5345"] 2017-06-19 14:27:53.225367 7f5d7e9a6700 10 mds.0.server reply_client_request -13 ((13) Permission denied) client_request(client.366267909:39615913 lookup #100017975f1/0442815e1ce9e96d66f49369877aeedd24db50 2017-06-19 14:27:53.224716) v3
Full log with debug_mds=20 is attached.
The client has path restricted mds caps, so we initially suspected #17858. But even with unrestricted mds caps the permission denied persists.
However, if we stat from a second host (which succeeds), then the original client A is able to stat the file without permission denied.
ceph-fuse and ceph-mds are version 10.2.7.
Happy to provide more logs if needed.
Files
Updated by Dan van der Ster almost 7 years ago
I should mention that while client A was a user that has a path-restricted mds cap, the client B that "fixes" the EPERM is client.admin with unrestricted mds caps. I didn't try using his restricted cap to stat from a second machine.
Updated by Dan van der Ster almost 7 years ago
I've confirmed that none of these help resolve these EPERM files:
- restart the ceph-fuse on client A
- mount with fuse_disable_pagecache = 1
- mount with fuse_default_permissions = 0
Updated by Dan van der Ster almost 7 years ago
Ahh so it is related to path-restricted cap.
I tried as above with client B having the same client caps -- didn't fix the EPERM.
But when I remove the path-restricted cap, then remount, then the EPERM files can be stat'd.
Could it be that the fix for #17858 is incomplete?
Updated by Dan van der Ster almost 7 years ago
Yup, in this case the diri is_stray (it looks like this
check_access stray_prior_path #10000ee3d6f/tmp_obj_x3Ncq5*
So it doesn't match the path prefix and cehck_access returns EPERM.
Relevant check is here, but I'm not sure how to turn #10000ee3d6f/tmp_obj_x3Ncq5 into the actual path.
int Session::check_access(CInode *in, unsigned mask, int caller_uid, int caller_gid, int new_uid, int new_gid) { string path; CInode *diri = NULL; if (!in->is_base()) diri = in->get_projected_parent_dn()->get_dir()->get_inode(); if (diri && diri->is_stray()){ path = in->get_projected_inode()->stray_prior_path; dout(20) << __func__ << " stray_prior_path " << path << dendl; } else { in->make_path_string(path, true); dout(20) << __func__ << " path " << path << dendl; }
path = in->get_projected_inode()->stray_prior_path; seems incorrect?
Updated by John Spray almost 7 years ago
- Assignee changed from John Spray to Zheng Yan
Updated by Zheng Yan almost 7 years ago
- Status changed from New to Fix Under Review
Updated by Dan van der Ster almost 7 years ago
Thanks for this patch. It seems to fix the problem for our users.
Updated by John Spray almost 7 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Nathan Cutler almost 7 years ago
- Copied to Backport #20403: jewel: cephfs permission denied until second client accesses file added
Updated by Nathan Cutler almost 7 years ago
- Copied to Backport #20404: kraken: cephfs permission denied until second client accesses file added
Updated by Nathan Cutler over 6 years ago
- Status changed from Pending Backport to Resolved