Project

General

Profile

Bug #12653

fuse mounted file systems fails SAMBA CTDB ping_pong rw test with v9.0.2

Added by Eric Eastman over 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Correctness/Safety
Target version:
-
Start date:
08/10/2015
Due date:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
ceph-fuse
Labels (FS):
Pull request ID:

Description

While working with the SAMBA CTDB ping_pong test and the Ceph file system, I noticed that file systems mounted using fuse and the kernel gives different results. According to the SAMBA documentation the results are correct for the kernel mount and incorrect for the fuse mount.

Using fuse to mount /cephfs on my 3 client systems, I then start the ping_pong test on each, and then captured the output while all 3 ping_pong tests are running:

root@ede-c1-gw03:/home/keeper# ./ping_pong -rw /cephfs/zzz 4
data increment = 1
^C 485 locks/sec

root@ede-c1-gw02:/home/keeper# ./ping_pong -rw /cephfs/zzz 4
data increment = 1
^C 358 locks/sec

root@ede-c1-gw01:/home/keeper# ./ping_pong -rw /cephfs/zzz 4
data increment = 1
^C 161 locks/sec

With the kernel mount of /cephfs

root@ede-c1-gw01:/home/keeper# ./ping_pong -rw /cephfs/zzz 4
data increment = 1
data increment = 2
data increment = 3
163 locks/sec

root@ede-c1-gw02:/home/keeper# ./ping_pong -rw /cephfs/zzz 4
data increment = 2
data increment = 3
167 locks/sec

root@ede-c1-gw03:/home/keeper# ./ping_pong -rw /cephfs/zzz 4
data increment = 3
159 locks/sec

The output for the kernel mounted /cephfs is correct as the data count increments with each additional system, which is correct per the ping_pong documentation.

Details on the test systems. All Ceph cluster nodes including the 3 ceph file system clients are running Ubuntu Trusty 14.04 with Ceph version:
ceph -v
ceph version 9.0.2 (be422c8f5b494c77ebcf0f7b95e5d728ecacb7f0)

Linux kernel version:
cat /proc/version
Linux version 4.1.3-040103-generic (kernel@gloin) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04) ) #201507220129 SMP Wed Jul 22 01:31:38 UTC 2015

I tested both fuse 2.9.2 shipped with Trusty and then tried 2.9.4 taken from the Ubuntu Wili distribution. They both had the same failing results.

Trusty version of fuse
dpkg -l | grep fuse
ii ceph-fuse 9.0.2-1trusty amd64 FUSE-based client for the Ceph distributed file system
ii fuse 2.9.2-4ubuntu4 amd64 Filesystem in Userspace
ii libfuse2:amd64 2.9.2-4ubuntu4 amd64 Filesystem in Userspace (library)

Wili version of fuse
dpkg -l | grep fuse
ii ceph-fuse 9.0.2-1trusty amd64 FUSE-based client for the Ceph distributed file system
ii fuse 2.9.4-1ubuntu1 amd64 Filesystem in Userspace
ii libfuse2:amd64 2.9.4-1ubuntu1 amd64 Filesystem in Userspace (library)

fstab entry for fuse mounts:
id=cephfs,keyring=/etc/ceph/client.cephfs.keyring /cephfs fuse.ceph noatime,_netdev,noauto 0 0

fstab entry for kernel mounts:
10.14.2.11,10.14.2.12,10.14.2.13:/ /cephfs ceph name=cephfs,secretfile=/etc/ceph/client.cephfs,noatime,_netdev 0 0

Document on ping_pong test: https://wiki.samba.org/index.php/Ping_pong

I have attached the ping_ping test exactable and source.

ping_pong (13.6 KB) Eric Eastman, 08/10/2015 03:39 AM

ping_pong.c View (3.34 KB) Eric Eastman, 08/10/2015 03:39 AM


Related issues

Copied to fs - Backport #16320: jewel: fs: fuse mounted file systems fails SAMBA CTDB ping_pong rw test with v9.0.2 Resolved

History

#1 Updated by Eric Eastman over 3 years ago

#2 Updated by Zheng Yan over 3 years ago

  • Status changed from New to Verified

This is a ceph-fuse bug, running the test directly on ceph-fuse mount can reproduce the bug. The bug is because that linux kernel cache data in page cache, modifying file on one host does not invalidate data in other hosts' page cache.

https://github.com/ceph/ceph/pull/5521

#3 Updated by Greg Farnum over 3 years ago

  • Status changed from Verified to Need Review

#4 Updated by John Spray almost 3 years ago

  • Status changed from Need Review to Pending Backport
  • Backport set to jewel

#5 Updated by Nathan Cutler almost 3 years ago

  • Copied to Backport #16320: jewel: fs: fuse mounted file systems fails SAMBA CTDB ping_pong rw test with v9.0.2 added

#6 Updated by Greg Farnum over 2 years ago

  • Category set to Correctness/Safety
  • Component(FS) ceph-fuse added

#7 Updated by Loic Dachary over 2 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF