Project

General

Profile

Actions

Bug #20340

closed

cephfs permission denied until second client accesses file

Added by Dan van der Ster almost 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
jewel, kraken
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
ceph-fuse
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Here is a file that client A gets permission denied during stat:

[root@aijens151 ~]# stat /mnt/puppet/aijens151.cern.ch/clone/hostgroups/paas/fix_lint/.git/objects/47/0442815e1ce9e96d66f49369877aeedd24db50
stat: cannot stat ‘/mnt/puppet/aijens151.cern.ch/clone/hostgroups/paas/fix_lint/.git/objects/47/0442815e1ce9e96d66f49369877aeedd24db50’: Permission denied

On the (single active) MDS, the permission denied looks like this:

2017-06-19 14:27:53.225362 7f5d7e9a6700 20 Session check_access stray_prior_path #100017975f1/tmp_obj_rFEkRh
2017-06-19 14:27:53.225363 7f5d7e9a6700 10 MDSAuthCap is_capable inode(path /100017975f1/tmp_obj_rFEkRh owner 993:991 mode 0100444) by caller 0:0 mask 1 new 0:0 cap: MDSAuthCaps[allow rw path="/volumes/_nogroup/4722edf2-a761-4189-b925-dbf2d44c5345"]
2017-06-19 14:27:53.225367 7f5d7e9a6700 10 mds.0.server reply_client_request -13 ((13) Permission denied) client_request(client.366267909:39615913 lookup #100017975f1/0442815e1ce9e96d66f49369877aeedd24db50 2017-06-19 14:27:53.224716) v3

Full log with debug_mds=20 is attached.

The client has path restricted mds caps, so we initially suspected #17858. But even with unrestricted mds caps the permission denied persists.

However, if we stat from a second host (which succeeds), then the original client A is able to stat the file without permission denied.

ceph-fuse and ceph-mds are version 10.2.7.

Happy to provide more logs if needed.


Files

ceph-mds.cephdwightmds1.log.gz (29.2 KB) ceph-mds.cephdwightmds1.log.gz Dan van der Ster, 06/19/2017 01:13 PM

Related issues 2 (0 open2 closed)

Copied to CephFS - Backport #20403: jewel: cephfs permission denied until second client accesses fileResolvedWei-Chung ChengActions
Copied to CephFS - Backport #20404: kraken: cephfs permission denied until second client accesses fileRejectedActions
Actions #1

Updated by Dan van der Ster almost 7 years ago

I should mention that while client A was a user that has a path-restricted mds cap, the client B that "fixes" the EPERM is client.admin with unrestricted mds caps. I didn't try using his restricted cap to stat from a second machine.

Actions #2

Updated by Dan van der Ster almost 7 years ago

I've confirmed that none of these help resolve these EPERM files:

  • restart the ceph-fuse on client A
  • mount with fuse_disable_pagecache = 1
  • mount with fuse_default_permissions = 0
Actions #3

Updated by Dan van der Ster almost 7 years ago

Ahh so it is related to path-restricted cap.
I tried as above with client B having the same client caps -- didn't fix the EPERM.

But when I remove the path-restricted cap, then remount, then the EPERM files can be stat'd.

Could it be that the fix for #17858 is incomplete?

Actions #4

Updated by Dan van der Ster almost 7 years ago

Yup, in this case the diri is_stray (it looks like this

check_access stray_prior_path #10000ee3d6f/tmp_obj_x3Ncq5*

So it doesn't match the path prefix and cehck_access returns EPERM.

Relevant check is here, but I'm not sure how to turn #10000ee3d6f/tmp_obj_x3Ncq5 into the actual path.

int Session::check_access(CInode *in, unsigned mask,
                          int caller_uid, int caller_gid,
                          int new_uid, int new_gid)
{
  string path;
  CInode *diri = NULL;
  if (!in->is_base())
    diri = in->get_projected_parent_dn()->get_dir()->get_inode();
  if (diri && diri->is_stray()){
    path = in->get_projected_inode()->stray_prior_path;
    dout(20) << __func__ << " stray_prior_path " << path << dendl;
  } else {
    in->make_path_string(path, true);
    dout(20) << __func__ << " path " << path << dendl;
  }

path = in->get_projected_inode()->stray_prior_path; seems incorrect?

Actions #5

Updated by John Spray almost 7 years ago

  • Assignee changed from John Spray to Zheng Yan
Actions #6

Updated by John Spray almost 7 years ago

  • Backport set to jewel, kraken
Actions #7

Updated by Zheng Yan almost 7 years ago

  • Status changed from New to Fix Under Review
Actions #8

Updated by Dan van der Ster almost 7 years ago

Thanks for this patch. It seems to fix the problem for our users.

Actions #9

Updated by John Spray almost 7 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #10

Updated by Nathan Cutler almost 7 years ago

  • Copied to Backport #20403: jewel: cephfs permission denied until second client accesses file added
Actions #11

Updated by Nathan Cutler almost 7 years ago

  • Copied to Backport #20404: kraken: cephfs permission denied until second client accesses file added
Actions #12

Updated by Nathan Cutler over 6 years ago

  • Status changed from Pending Backport to Resolved
Actions

Also available in: Atom PDF