Bug #14736
closedKernel cephfs bind-mounts break if parent dir modified elsewhere
0%
Description
ISSUE
If you have used mount with the --bind option to overlay one cephfs location on another, then creating any file in the parent directory that holds the mountpoint with lead to a "Device or resource busy" error on any operation on that mountpoint, including any attempt to unmount it. Recoverable only by rebooting.
Syslog will report something like this whenever you try to do anything on that previously-accessible mountpoint:
ceph: splice_dentry error -16 ffff88003d2a2d98 inode ffff88003d29f988 ino 100000b0022.fffffffffffffffe
SCOPE
This behaviour exhibits itself in the cephfs kernel client, but not with the FUSE client.
HOW TO REPRODUCE
The following shell sequence assumes two nodes, having both mounted the same cephfs file-system on /data, via the kernel driver. It assumes you are root:
[ON SERVER 1]
mkdir -p /data/source
echo ok > /data/source/file_in_source
mkdir -p /data/target/mountpoint/
mount --bind /data/source /data/target/mountpoint
cat /data/target/mountpoint/file_in_source
- you should have received the output:
- ok
[ON SERVER 2, WITH THE SAME /data CEPHFS:]
mount --bind /data/source /data/target/mountpoint
echo ok > /data/target/file_in_parent
[BACK ON SERVER 1]
cat /data/target/mountpoint/file_in_source
- This time, you will receive an error:
- cat: /data/target/mountpoint/file_in_source: Device or resource busy
- /data/target/mountpoint is now unusable and, indeed, unmountable
Updated by Zheng Yan about 8 years ago
- Status changed from New to 12
- Tags changed from fuse to kcephfs
when creating /data/target/file_in_parent on server 2, MDS revokes directory (/data/target) inode's Fs capability from server 1. This causes dentry (data/target/mountpoint) become invalid. When accessing /data/target/mountpoint/file_in_source again, The kernel invalidates dentry (data/target/mountpoint) and causes this issue.
I think other network filesystems suffer this issue too. So far I have no idea how to fix this, please avoid using directory in cephfs as mount point.
Updated by Greg Farnum about 8 years ago
Just to be clear, you mean don't use a cephfs directory as the location for mounting another FS, right? And it's perfectly okay to mount /ceph/my/mount/point in to your local machine. :)
Updated by Nick Mailer about 8 years ago
Greg, we're talking about --bind mounting, not mounting a whole new filesystem.
So, mount --bind /ceph/some/directory /ceph/another/directory
And Zheng, this does work fine with other network filesystems. For example, it works fine on NFS and, more significantly, it works fine using the FUSE version of CephFS.
Updated by Zheng Yan about 8 years ago
Greg Farnum wrote:
Just to be clear, you mean don't use a cephfs directory as the location for mounting another FS, right? And it's perfectly okay to mount /ceph/my/mount/point in to your local machine. :)
yes
Updated by Zheng Yan about 8 years ago
Nick Mailer wrote:
Greg, we're talking about --bind mounting, not mounting a whole new filesystem.
So, mount --bind /ceph/some/directory /ceph/another/directory
And Zheng, this does work fine with other network filesystems. For example, it works fine on NFS and, more significantly, it works fine using the FUSE version of CephFS.
I checked the code again. There is some room for improvement. But using directory in cephfs as mount point is still dangerous,because we can delete the directory from other host.
Updated by Zheng Yan about 8 years ago
upstream commit 200fd27c8fa2ba8bb4529033967b69a7cbfa2c2e