Bug #44821
openCan not move directory when CephFS is lower layer for OverlayFS
0%
Description
When CephFS is used as a lower layer for OverlayFS and the redirect_dir=on option is supplied to OverlayFS, trying to move a file in the merged mount point fails with:
mv: cannot move 'merged/foo' to 'merged/bar': Operation not supported
This happens with the 4.15.0-91-generic Ubuntu kernel and with the 14.2.8-1bonic FUSE package.
Steps to reproduce:
mkdir /mnt/cephfs/test_dir/foo
cd /dev/shm
mkdir upper work merged
sudo mount -t overlay -o lowerdir=/mnt/cephfs/test_dir,upperdir=upper,workdir=work,redirect_dir=on overlay merged
mv merged/foo merged/bar
Ideally we would like to use this merged mount as the lower for even more overlayfs mount points.
Updated by Robert LeBlanc about 4 years ago
Sorry, the error is when moving a directory, not specifically a file. The steps to reproduce is correct.
Updated by Greg Farnum about 4 years ago
This sounds like an overlayfs issue. (Unless there are CephFS quotas involved? https://tracker.ceph.com/issues/44791)
Is there something specific CephFS would need to do to support it?
Updated by Greg Farnum about 4 years ago
- Status changed from New to Need More Info
Updated by Robert LeBlanc about 4 years ago
Ceph seems to be missing something in terms of xattrs. Here are some notes I took.
OverlayFS uses xattrs to determine when a directory needs to be promoted to the upper layer. It tries to access the xattr of the lower file and then croaks. From what I understand, the xattr is only stored in the upper layer, so even though tmpfs supports xattrs, it still fails trying to access the xattr from the lower layer. The fuse-xattr program layers xattrs onto NFS so that overlayFS works.
So I'm not sure if CephFS is returning differently than ext4/xfs/etc when a requested xattr is asked for and does not exist which causes overlayfs to not be happy.
Updated by Luis Henriques over 3 years ago
Could it be that you're using tmpfs without xattrs support? You can check this in your kernel config, if it has CONFIG_TMPFS_XATTR enabled.
Updated by Luis Henriques over 3 years ago
After some more digging with a different report [1], I found out that the problem may actually be related with the fact that you're using an old kernel that lists all the vxattrs in listxattr(). Basically, overlayfs will try to copy all the vxattrs and that will fail.
This ix fixed by commit e09580b343aa ("ceph: don't list vxattrs in listxattr()"), in 5.4.
[1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/BIOBKO67OFYMZ245RIH4XG6BANW7HCAY/