Project

General

Profile

Bug #20340

cephfs permission denied until second client accesses file

Added by Dan van der Ster 5 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
06/19/2017
Due date:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
jewel, kraken
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
jewel
Component(FS):
ceph-fuse
Needs Doc:
No

Description

Here is a file that client A gets permission denied during stat:

[root@aijens151 ~]# stat /mnt/puppet/aijens151.cern.ch/clone/hostgroups/paas/fix_lint/.git/objects/47/0442815e1ce9e96d66f49369877aeedd24db50
stat: cannot stat ‘/mnt/puppet/aijens151.cern.ch/clone/hostgroups/paas/fix_lint/.git/objects/47/0442815e1ce9e96d66f49369877aeedd24db50’: Permission denied

On the (single active) MDS, the permission denied looks like this:

2017-06-19 14:27:53.225362 7f5d7e9a6700 20 Session check_access stray_prior_path #100017975f1/tmp_obj_rFEkRh
2017-06-19 14:27:53.225363 7f5d7e9a6700 10 MDSAuthCap is_capable inode(path /100017975f1/tmp_obj_rFEkRh owner 993:991 mode 0100444) by caller 0:0 mask 1 new 0:0 cap: MDSAuthCaps[allow rw path="/volumes/_nogroup/4722edf2-a761-4189-b925-dbf2d44c5345"]
2017-06-19 14:27:53.225367 7f5d7e9a6700 10 mds.0.server reply_client_request -13 ((13) Permission denied) client_request(client.366267909:39615913 lookup #100017975f1/0442815e1ce9e96d66f49369877aeedd24db50 2017-06-19 14:27:53.224716) v3

Full log with debug_mds=20 is attached.

The client has path restricted mds caps, so we initially suspected #17858. But even with unrestricted mds caps the permission denied persists.

However, if we stat from a second host (which succeeds), then the original client A is able to stat the file without permission denied.

ceph-fuse and ceph-mds are version 10.2.7.

Happy to provide more logs if needed.

ceph-mds.cephdwightmds1.log.gz (29.2 KB) Dan van der Ster, 06/19/2017 01:13 PM


Related issues

Copied to fs - Backport #20403: jewel: cephfs permission denied until second client accesses file Resolved
Copied to fs - Backport #20404: kraken: cephfs permission denied until second client accesses file Rejected

History

#1 Updated by Dan van der Ster 5 months ago

I should mention that while client A was a user that has a path-restricted mds cap, the client B that "fixes" the EPERM is client.admin with unrestricted mds caps. I didn't try using his restricted cap to stat from a second machine.

#2 Updated by Dan van der Ster 5 months ago

I've confirmed that none of these help resolve these EPERM files:

  • restart the ceph-fuse on client A
  • mount with fuse_disable_pagecache = 1
  • mount with fuse_default_permissions = 0

#3 Updated by Dan van der Ster 5 months ago

Ahh so it is related to path-restricted cap.
I tried as above with client B having the same client caps -- didn't fix the EPERM.

But when I remove the path-restricted cap, then remount, then the EPERM files can be stat'd.

Could it be that the fix for #17858 is incomplete?

#4 Updated by Dan van der Ster 5 months ago

Yup, in this case the diri is_stray (it looks like this

check_access stray_prior_path #10000ee3d6f/tmp_obj_x3Ncq5*

So it doesn't match the path prefix and cehck_access returns EPERM.

Relevant check is here, but I'm not sure how to turn #10000ee3d6f/tmp_obj_x3Ncq5 into the actual path.

int Session::check_access(CInode *in, unsigned mask,
                          int caller_uid, int caller_gid,
                          int new_uid, int new_gid)
{
  string path;
  CInode *diri = NULL;
  if (!in->is_base())
    diri = in->get_projected_parent_dn()->get_dir()->get_inode();
  if (diri && diri->is_stray()){
    path = in->get_projected_inode()->stray_prior_path;
    dout(20) << __func__ << " stray_prior_path " << path << dendl;
  } else {
    in->make_path_string(path, true);
    dout(20) << __func__ << " path " << path << dendl;
  }

path = in->get_projected_inode()->stray_prior_path; seems incorrect?

#5 Updated by John Spray 5 months ago

  • Assignee changed from John Spray to Zheng Yan

#6 Updated by John Spray 5 months ago

  • Backport set to jewel, kraken

#7 Updated by Zheng Yan 5 months ago

  • Status changed from New to Need Review

#8 Updated by Dan van der Ster 5 months ago

Thanks for this patch. It seems to fix the problem for our users.

#9 Updated by John Spray 5 months ago

  • Status changed from Need Review to Pending Backport

#10 Updated by Nathan Cutler 5 months ago

  • Copied to Backport #20403: jewel: cephfs permission denied until second client accesses file added

#11 Updated by Nathan Cutler 5 months ago

  • Copied to Backport #20404: kraken: cephfs permission denied until second client accesses file added

#12 Updated by Nathan Cutler 3 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF