Ceph » CephFS

While working with the SAMBA CTDB ping_pong test and the Ceph file system, I noticed that file systems mounted using fuse and the kernel gives different results. According to the SAMBA documentation the results are correct for the kernel mount and incorrect for the fuse mount.

Using fuse to mount /cephfs on my 3 client systems, I then start the ping_pong test on each, and then captured the output while all 3 ping_pong tests are running:

root@ede-c1-gw03:/home/keeper# ./ping_pong -rw /cephfs/zzz 4
data increment = 1
^C 485 locks/sec

root@ede-c1-gw02:/home/keeper# ./ping_pong -rw /cephfs/zzz 4
data increment = 1
^C 358 locks/sec

root@ede-c1-gw01:/home/keeper# ./ping_pong -rw /cephfs/zzz 4
data increment = 1
^C 161 locks/sec

With the kernel mount of /cephfs

root@ede-c1-gw01:/home/keeper# ./ping_pong -rw /cephfs/zzz 4
data increment = 1
data increment = 2
data increment = 3
163 locks/sec

root@ede-c1-gw02:/home/keeper# ./ping_pong -rw /cephfs/zzz 4
data increment = 2
data increment = 3
167 locks/sec

root@ede-c1-gw03:/home/keeper# ./ping_pong -rw /cephfs/zzz 4
data increment = 3
159 locks/sec

The output for the kernel mounted /cephfs is correct as the data count increments with each additional system, which is correct per the ping_pong documentation.

Details on the test systems. All Ceph cluster nodes including the 3 ceph file system clients are running Ubuntu Trusty 14.04 with Ceph version:
ceph -v
ceph version 9.0.2 (be422c8f5b494c77ebcf0f7b95e5d728ecacb7f0)

Linux kernel version:
cat /proc/version
Linux version 4.1.3-040103-generic (kernel@gloin) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04) ) #201507220129 SMP Wed Jul 22 01:31:38 UTC 2015

I tested both fuse 2.9.2 shipped with Trusty and then tried 2.9.4 taken from the Ubuntu Wili distribution. They both had the same failing results.

Trusty version of fuse
dpkg -l | grep fuse
ii ceph-fuse 9.0.2-1trusty amd64 FUSE-based client for the Ceph distributed file system
ii fuse 2.9.2-4ubuntu4 amd64 Filesystem in Userspace
ii libfuse2:amd64 2.9.2-4ubuntu4 amd64 Filesystem in Userspace (library)

Wili version of fuse
dpkg -l | grep fuse
ii ceph-fuse 9.0.2-1trusty amd64 FUSE-based client for the Ceph distributed file system
ii fuse 2.9.4-1ubuntu1 amd64 Filesystem in Userspace
ii libfuse2:amd64 2.9.4-1ubuntu1 amd64 Filesystem in Userspace (library)

fstab entry for fuse mounts:
id=cephfs,keyring=/etc/ceph/client.cephfs.keyring /cephfs fuse.ceph noatime,_netdev,noauto 0 0

fstab entry for kernel mounts:
10.14.2.11,10.14.2.12,10.14.2.13:/ /cephfs ceph name=cephfs,secretfile=/etc/ceph/client.cephfs,noatime,_netdev 0 0

Document on ping_pong test: https://wiki.samba.org/index.php/Ping_pong

I have attached the ping_ping test exactable and source.