Ceph » CephFS

Since upgrading from Nautilus (14.2.21) to Pacific (16.2.7) ceph-fuse shows a rather unexpected and unusual behavior when a directory has been set to mode 0600. This problem did not exist in versions of ceph-fuse up to 14.2.21. As we have never used Octopus it is unclear when exactly the problem has been introduced. As we have one server left still running ceph-fuse 14.2.21 while all others have been upgraded to Pacific we know that using the same underlying CephFS (16.2.7) the problem only occurs when using ceph-fuse 16.2.7 while when using the old version of ceph-fuse everything still works as expected.

This behavior breaks various software packages using access mode 0600 and at the same time makes it impossible to backup these directories and their contents which may end up in critical data losses.

Run the following simple test (using sudo or as root) on a CephFS using ceph-fuse 16.2.7 and compare it to the behavior when doing the same using Linux (ext3/4), macOS (BSD) or a ceph file system mounted on Linux using ceph-fuse 14.2.21. The difference is obvious.

# Execute the following commands in a directory in which you have r/w access

mkdir tstdir
echo "abc" > tstdir/tstfile
sudo chmod 600 tstdir

ls -ld tstdir # up till here everything works as expected
macOS:        drw-------  3 user group  96 12 Apr 11:12 tstdir/
Linux ext4:   drw------- 2 user group 4096 Apr 12 11:19 tstdir
ceph-fuse 16: drw------- 2 user group 4 Apr 12 11:13 tstdir/

sudo ls -l tstdir/tstfile
macOS:        -rw-r--r--  1 user  group  4 12 Apr 11:12 tstdir/tstfile
Linux ext4:   -rw-r--r-- 1 user group 4 Apr 12 11:19 tstdir/tstfile
ceph-fuse 16: ls: cannot access 'tstdir/tstfile': Permission denied

sudo ls -l tstdir
macOS:        -rw-r--r--  1 user  group  4 12 Apr 11:12 tstdir/tstfile
Linux ext4:   -rw-r--r-- 1 user group 4 Apr 12 11:19 tstdir/tstfile
ceph-fuse 16: ls: cannot access 'tstdir/tstfile': Permission denied
              total 0
              -????????? ? ? ? ?            ? tstfile

sudo cat tstdir/tstfile
macOS:        abc
Linux ext4:   abc
ceph-fuse 16: cat: tstdir/tstfile: Permission denied

sudo rm -rf tstdir/
macOS:        (removes dir)
Linux ext4:   (removes dir)
ceph-fuse 16: rm: cannot remove 'tstdir/tstfile': Permission denied

Here is some information about the environment

1. ceph --version
   ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)

1. uname -a
   Linux server 5.10.0-13-amd64 #1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux

1. dpkg -l | grep ceph-fuse
   ii ceph-fuse 16.2.7-1~bpo11+1 amd64 FUSE-based client for the Ceph distributed file system

1. mount | grep /cephfs
   ceph-fuse on /cephfs type fuse.ceph-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)