Project

General

Profile

Actions
Copy link

Bug #12189

closed

Editing / Creating files fails for NFS-over-CephFS on EC pool with cache tier

Added by Burkhard Linke almost 9 years ago. Updated over 8 years ago.

Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
Reviewed:
Affected Versions:
Ceph - v0.94.2
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Ubuntu 14.04, Kernel 3.13.0-55-generic
Standard kernel-based NFS server
Ceph Hammer release
~# ceph version
ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)

Mount point for cephfs via fstab:
client_mountpoint=/volumes /srv/ceph fuse.ceph defaults,_netdev 0 0

~# mount | grep ceph
ceph-fuse on /srv/ceph type fuse.ceph-fuse (rw,nosuid,nodev,allow_other,default_permissions)

Mount point is exported to kerberized desktop machines in another network:
~# cat /etc/exports | grep ceph
  1. export ceph to desktop machines only
    /srv/ceph XX.XX.XX.XX/YY

On client machine export CephFS is mounted as /ceph:
~$ mount | grep ceph
nfs-homes:/srv/ceph on /ceph type nfs (rw,noatime,fsc,nfsvers=4,sec=krb5p,intr,ac,sloppy,addr=XX.XX.XX.XX,clientaddr=XX.XX.XX.XX)

CephFS uses three data pools:
~# ceph fs ls
name: cephfs, metadata pool: cephfs_test_metadata, data pools: [cephfs_test_data cephfs_two_rep_data ec_ssd_cache ]

Two pools are replicated pools, one pool is cache tier of an erasure coding pool. The default pool is the EC pool (resp. its cache tier):
~# getfattr -n ceph.dir.layout /srv/ceph/
getfattr: Removing leading '/' from absolute path names
  1. file: srv/ceph/
    ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=ec_ssd_cache"

# getfattr -n ceph.dir.layout /srv/ceph/adm/temp/test/
/srv/ceph/adm/temp/test/: ceph.dir.layout: No such attribute
# getfattr -n ceph.dir.layout /srv/ceph/adm/temp/
/srv/ceph/adm/temp/: ceph.dir.layout: No such attribute
~# getfattr -n ceph.dir.layout /srv/ceph/adm/
/srv/ceph/adm/: ceph.dir.layout: No such attribute

Creating files and/or editing them on the server works:

:/srv/ceph/adm/temp/test$ dd if=/dev/zero of=test bs=1 count=10
10+0 records in
10+0 records out
10 bytes (10 B) copied, 0.0046936 s, 2.1 kB/s

Creating a file in the same way on a NFS client also works:

:/ceph/adm/temp/test$ dd if=/dev/zero of=test2 bs=1 count=10
10+0 records in
10+0 records out
10 bytes (10 B) copied, 0.0332668 s, 0.3 kB/s

Editing the file on the server is also ok:

:/srv/ceph/adm/temp/test$ vi test
blinke@waas:/srv/ceph/adm/temp/test$ ls al
total 2
drwxr-xr-x 1 blinke cb 20 Jun 30 16:25 .
drwxrwxrwt 1 blinke support 20733315 Jun 30 16:11 ..
-rw-r--r- 1 blinke cb 10 Jun 30 16:23 test
rw-r--r- 1 blinke cb 10 Jun 30 16:24 test2

Editing the file on the NFS client FAILS:

:/ceph/adm/temp/test$ vi test2

E325: ATTENTION
Found a swap file by the name ".test2.swp"
owned by: blinke dated: Mon Dec 11 03:21:47 1972
[cannot be opened]
While opening file "test2"
dated: Tue Jun 30 16:24:29 2015
NEWER than swap file!

(1) Another program may be editing the same file. If this is the case,
be careful not to end up with two different instances of the same
file when making changes. Quit, or continue with caution.
(2) An edit session for this file crashed.
If this is the case, use ":recover" or "vim -r test2"
to recover the changes (see ":help recovery").
If you did this already, delete the swap file ".test2.swp"
to avoid this message.

Swap file ".test2.swp" already exists!
[O]pen Read-Only, (E)dit anyway, (R)ecover, (D)elete it, (Q)uit, (A)bort: -> q

:/ceph/adm/temp/test$ vi test2
blinke@fb08-bcf-pc01:/ceph/adm/temp/test$ ls al
total 2
drwxr-xr-x 1 blinke cb 20 Jun 30 16:26 .
drwxrwxrwt 1 blinke support 20733315 Jun 30 16:11 ..
-rw-r--r- 1 blinke cb 10 Jun 30 16:23 test
rw-r--r- 1 blinke cb 10 Jun 30 16:24 test2
---------- 1 blinke cb 0 Dec 11 1972 .test2.swo
---------- 1 blinke cb 0 Dec 11 1972 .test2.swp

Editing a files stored on a replicated pool is OK

Copying a file to the directory on the server is OK:
:/srv/ceph/adm/temp/test$ cp ~/wf1.out .
:/srv/ceph/adm/temp/test$ rm wf1.out

Copying a file to the directory on the client FAILS:
:/ceph/adm/temp/test$ cp ~/wf1.out .
cp: cannot create regular file './wf1.out': Permission denied
:/ceph/adm/temp/test$ ls al
total 2
drwxr-xr-x 1 blinke cb 20 Jun 30 16:30 .
drwxrwxrwt 1 blinke support 20728068 Jun 30 16:11 ..
-rw-r--r- 1 blinke cb 10 Jun 30 16:23 test
rw-r--r- 1 blinke cb 10 Jun 30 16:24 test2
---------- 1 blinke cb 0 Dec 11 1972 .test2.swo
---------- 1 blinke cb 0 Dec 11 1972 .test2.swp
---------- 1 blinke cb 0 Dec 11 1972 wf1.out

It is created as empty file....

Actions
Copy link

Also available in: Atom PDF