Project

General

Profile

Bug #55313

Unexpected file access behavior using ceph-fuse

Added by Matthias Aebi 8 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Category:
Correctness/Safety
Target version:
% Done:

100%

Source:
Tags:
Backport:
quincy,pacific
Regression:
Yes
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
ceph-fuse
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Since upgrading from Nautilus (14.2.21) to Pacific (16.2.7) ceph-fuse shows a rather unexpected and unusual behavior when a directory has been set to mode 0600. This problem did not exist in versions of ceph-fuse up to 14.2.21. As we have never used Octopus it is unclear when exactly the problem has been introduced. As we have one server left still running ceph-fuse 14.2.21 while all others have been upgraded to Pacific we know that using the same underlying CephFS (16.2.7) the problem only occurs when using ceph-fuse 16.2.7 while when using the old version of ceph-fuse everything still works as expected.

This behavior breaks various software packages using access mode 0600 and at the same time makes it impossible to backup these directories and their contents which may end up in critical data losses.

Run the following simple test (using sudo or as root) on a CephFS using ceph-fuse 16.2.7 and compare it to the behavior when doing the same using Linux (ext3/4), macOS (BSD) or a ceph file system mounted on Linux using ceph-fuse 14.2.21. The difference is obvious.

# Execute the following commands in a directory in which you have r/w access

mkdir tstdir
echo "abc" > tstdir/tstfile
sudo chmod 600 tstdir

ls -ld tstdir # up till here everything works as expected
macOS:        drw-------  3 user group  96 12 Apr 11:12 tstdir/
Linux ext4:   drw------- 2 user group 4096 Apr 12 11:19 tstdir
ceph-fuse 16: drw------- 2 user group 4 Apr 12 11:13 tstdir/

sudo ls -l tstdir/tstfile
macOS:        -rw-r--r--  1 user  group  4 12 Apr 11:12 tstdir/tstfile
Linux ext4:   -rw-r--r-- 1 user group 4 Apr 12 11:19 tstdir/tstfile
ceph-fuse 16: ls: cannot access 'tstdir/tstfile': Permission denied

sudo ls -l tstdir
macOS:        -rw-r--r--  1 user  group  4 12 Apr 11:12 tstdir/tstfile
Linux ext4:   -rw-r--r-- 1 user group 4 Apr 12 11:19 tstdir/tstfile
ceph-fuse 16: ls: cannot access 'tstdir/tstfile': Permission denied
              total 0
              -????????? ? ? ? ?            ? tstfile

sudo cat tstdir/tstfile
macOS:        abc
Linux ext4:   abc
ceph-fuse 16: cat: tstdir/tstfile: Permission denied

sudo rm -rf tstdir/
macOS:        (removes dir)
Linux ext4:   (removes dir)
ceph-fuse 16: rm: cannot remove 'tstdir/tstfile': Permission denied

Here is some information about the environment

  1. ceph --version
    ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
  1. uname -a
    Linux server 5.10.0-13-amd64 #1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux
  1. dpkg -l | grep ceph-fuse
    ii ceph-fuse 16.2.7-1~bpo11+1 amd64 FUSE-based client for the Ceph distributed file system
  1. mount | grep /cephfs
    ceph-fuse on /cephfs type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

Related issues

Copied to CephFS - Backport #55926: quincy: Unexpected file access behavior using ceph-fuse Resolved
Copied to CephFS - Backport #55927: pacific: Unexpected file access behavior using ceph-fuse Resolved

History

#1 Updated by Venky Shankar 7 months ago

  • Assignee set to Kotresh Hiremath Ravishankar
  • Target version set to v18.0.0
  • Backport set to quincy,pacific

Thanks for the report, Matthias. This seems straightforward to reproduce.

Kotresh, please take a look.

#2 Updated by Kotresh Hiremath Ravishankar 7 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 46078

#3 Updated by Kotresh Hiremath Ravishankar 7 months ago

Hi Matthias,

Quick workaround would be to set "fuse_default_permissions=true" but it might cost you performance.

#4 Updated by Matthias Aebi 7 months ago

Ok, thank you. I'll certainly give this a try. Besides some cost in performance, does this have any impact on who might have access to which files/directories (except for fixing the issue described above)?

Does this option have to be set in /etc/fstab, does it go into /etc/ceph/ceph.conf (if yes, what is the right section to put this) or does it belong into any other configuration file? Thank you for a hint.

#5 Updated by Kotresh Hiremath Ravishankar 7 months ago

Matthias Aebi wrote:

Ok, thank you. I'll certainly give this a try. Besides some cost in performance, does this have any impact on who might have access to which files/directories (except for fixing the issue described above)?

I think no. It uses kernel-enforced permissions when the option is set to true

Does this option have to be set in /etc/fstab, does it go into /etc/ceph/ceph.conf (if yes, what is the right section to put this) or does it belong into any other configuration file? Thank you for a hint.

This should go in /etc/ceph/ceph.conf under the client section.

...
[client]
fuse default permissions = 1

Note that the option is marked with 'startup' flag. So existing mounts won't reflect it after adding into 'ceph.conf'.
New mounts would use that option.

#6 Updated by Matthias Aebi 7 months ago

Thanks, I can confirm that this works and as you mentioned does slow down file access. In our case, which is an rsnyc backup, the time needed for the job has increased by about 50% from 40 to 60 minutes. So it will certainly be good to get the problem solved at its root.

#7 Updated by Kotresh Hiremath Ravishankar 7 months ago

Yes, the PR is posted and would be backported.

#8 Updated by Venky Shankar 6 months ago

  • Status changed from Fix Under Review to Pending Backport

#9 Updated by Backport Bot 6 months ago

  • Copied to Backport #55926: quincy: Unexpected file access behavior using ceph-fuse added

#10 Updated by Backport Bot 6 months ago

  • Copied to Backport #55927: pacific: Unexpected file access behavior using ceph-fuse added

#11 Updated by Backport Bot 4 months ago

  • Tags set to backport_processed

#12 Updated by Konstantin Shalygin about 2 months ago

  • Status changed from Pending Backport to Resolved
  • % Done changed from 0 to 100
  • Tags deleted (backport_processed)

Also available in: Atom PDF