Ceph » CephFS

This works for me. mounted the same filesystem twice (without noshare option):

$ grep ceph /proc/self/mountinfo
238 61 0:37 / /mnt/cephfs rw,relatime shared:131 - ceph 192.168.1.80:6789:/ rw,name=admin,secret=<hidden>,acl
266 61 0:37 / /mnt/cephfs1 rw,relatime shared:138 - ceph 192.168.1.80:6789:/ rw,name=admin,secret=<hidden>,acl

The third column tells us the superblock's device number. It's the same here because it found the same superblock.

If I unmount and mount the second mount again, this time with -o noshare:

$ grep ceph /proc/self/mountinfo
238 61 0:37 / /mnt/cephfs rw,relatime shared:131 - ceph 192.168.1.80:6789:/ rw,name=admin,secret=<hidden>,acl
266 61 0:40 / /mnt/cephfs1 rw,relatime shared:138 - ceph 192.168.1.80:6789:/ rw,name=admin,secret=<hidden>,noshare,acl

...this time I get a different device number (because the superblock is different).

Note that this is using a v5.5.8 kernel, but this code hasn't change substantially in a long time, so I'd expect the same behavior in any kernel from the last few years.

What behavior were you expecting, and what kernel were you using to test this?

if I create two cephfs named:cephfs and zyli.
I want mount them using kernel mount:

mount -t ceph 10.192.35.164:/ /nas/cephfs -o name=admin,secretfile=/etc/ceph/ceph.client.key,mds_namespace=cephfs
mount -t ceph 10.192.35.164:/ /nas/zyli -o name=admin,secretfile=/etc/ceph/ceph.client.key,mds_namespace=zyli

i can see two mount info use "grep ceph /proc/self/mountinfo"

but i can not see two mountpoint use "df -h"

also i use ceph-fuse to mount them

ceph-fuse /nas/cephfs --client_mds_namespace cephfs
ceph-fuse /nas/zyli --client_mds_namespace zyli

i can see two mount info use "grep ceph /proc/self/mountinfo"

and i alse can see two mountpoint use "df -h"

On my machine (recent Fedora), 'df' works by reading /proc/self/mountinfo to get a list of mountpoints, and issuing stat() and statfs() calls for each.

If you see both entries in /proc/self/mountinfo then your df command may work differently than mine. You may want to strace the df command to get a better idea of what it's actually doing.

Jeff Layton wrote:

On my machine (recent Fedora), 'df' works by reading /proc/self/mountinfo to get a list of mountpoints, and issuing stat() and statfs() calls for each.

If you see both entries in /proc/self/mountinfo then your df command may work differently than mine. You may want to strace the df command to get a better idea of what it's actually doing.

I know what the problem is
i use kernel to mount cephfs like

mount -t ceph 10.192.35.164:/ /nas/cephfs -o name=admin,secretfile=/etc/ceph/ceph.client.key,mds_namespace=cephfs

then use 'stat' to get /nas/cephfs devicenum like

[root@node1 ~]# stat /nas/cephfs
File: ‘/nas/cephfs’
Size: 3 Blocks: 0 IO Block: 65536 directory
Device: 0h/0d Inode: 1 Links: 1
Access: (0777/drwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 1970-01-01 08:00:00.000000000 +0800
Modify: 2020-04-01 11:27:30.076626844 +0800
Change: 2020-04-01 15:41:50.505996328 +0800
Birth: -

the device num is 0.but in /proc/self/mountinfo devicenum is 43

[root@node1 ~]# cat /proc/self/mountinfo |grep /nas/cephfs
328 61 0:43 / /nas/cephfs rw,relatime shared:234 - ceph 10.193.55.180:/ rw,name=admin,secret=<hidden>,acl,wsize=16777216


and then i use ceph-fuse to mount cephfs.

ceph-fuse /nas/cephfs --client_mds_namespace cephfs

the device num is 43 in /proc/self/mountinfo and use stat command to get.

[root@node1 ~]# stat /nas/cephfs
File: ‘/nas/cephfs’
Size: 15938 Blocks: 1 IO Block: 4194304 directory
Device: 2bh/43d Inode: 1 Links: 5
Access: (0777/drwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 1970-01-01 08:00:00.000000000 +0800
Modify: 2020-04-01 11:27:30.076626844 +0800
Change: 2020-04-01 15:41:50.505996328 +0800
Birth: -

[root@node1 ~]# cat /proc/self/mountinfo |grep /nas/cephfs
328 61 0:43 / /nas/cephfs rw,nosuid,nodev,relatime shared:234 - fuse.ceph-fuse ceph-fuse rw,user_id=0,group_id=0,allow_other

and i also write a test program to get devicenum like

#include <sys/stat.h>
#include <unistd.h>
#include <stdio.h>

int main() {
struct stat buf;
stat("/nas/cephfs/",&buf);
printf("devnum = %ld\n",buf.st_dev);
return 0;
}

my question is why the devicenum is different between /proc/self/mountinfo and stat command when i use kernel to mount cephfs.

I don't see that at all with a more recent kernel (5.7.0-rc1). I'll have to spin up a centos box and see if it's reproducible there:

$ grep ceph-a !$
grep ceph-a /proc/self/mountinfo
337 60 0:38 / /mnt/ceph-a rw,relatime shared:182 - ceph 192.168.1.80:6789:/ rw,name=fs,secret=<hidden>,acl,mds_namespace=a

$ stat /mnt/ceph-a
  File: /mnt/ceph-a
  Size: 2             Blocks: 0          IO Block: 65536  directory
Device: 26h/38d    Inode: 1           Links: 4
Access: (1777/drwxrwxrwt)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:cephfs_t:s0
Access: 2020-04-13 14:19:52.120568463 -0400
Modify: 2020-04-08 08:57:51.202682232 -0400
Change: 2020-04-08 08:57:51.202682232 -0400
 Birth: 2020-03-24 14:57:51.722624114 -0400