Ceph » CephFS

~# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
126T 53823G 75664G 58.38
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
cephfs_test_data 7 918M 0 8764G 5614112
cephfs_test_metadata 8 36797k 0 833G 648204
cephfs_two_rep_data 12 28242G 21.79 13147G 9231148
ec_ssd_cache 18 388G 0.30 1250G 364092
cephfs_ec_data 19 10600G 8.18 17529G 5106582

:~# ceph osd dump | grep pool
pool 7 'cephfs_test_data' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 68813 flags hashpspool min_read_recency_for_promote 1 stripe_width 0
pool 8 'cephfs_test_metadata' replicated size 3 min_size 2 crush_ruleset 2 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 80938 flags hashpspool min_read_recency_for_promote 1 stripe_width 0
pool 12 'cephfs_two_rep_data' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 73815 flags hashpspool min_read_recency_for_promote 1 stripe_width 0
pool 18 'ec_ssd_cache' replicated size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 128 pgp_num 128 last_change 162775 flags hashpspool,incomplete_clones tier_of 19 cache_mode writeback target_bytes 500000000000 target_objects 1000000 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 3600s x1 stripe_width 0
pool 19 'cephfs_ec_data' erasure size 6 min_size 4 crush_ruleset 3 object_hash rjenkins pg_num 256 pgp_num 256 last_change 149589 lfor 149589 flags hashpspool tiers 18 read_tier 18 write_tier 18 stripe_width 4096

(removed other pools from output)

Replicated pools also seems to be affected:

On client:
:/ceph/test$ ls
:/ceph/test$ touch foo
:/ceph/test$ cp ~/wf1.out .
cp: cannot create regular file './wf1.out': Permission denied
blinke@fb08-bcf-pc01:/ceph/test$ ls al
total 1
drwxr-xr-x 1 blinke root 0 Jun 30 16:55 .
drwxr-xr-x 1 root root 30592876135254 Jun 30 16:53 ..
-rw-r--r- 1 blinke cb 0 Jun 30 16:55 foo

... some seconds later...

blinke@fb08-bcf-pc01:/ceph/test$ ls al
total 1
drwxr-xr-x 1 blinke root 0 Jun 30 16:55 .
drwxr-xr-x 1 root root 30592876135254 Jun 30 16:53 ..
-rw-r--r- 1 blinke cb 0 Jun 30 16:55 foo
---------- 1 blinke cb 0 Dec 16 1972 wf1.out

1. file: srv/ceph/test/
   ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_two_rep_data"

it's likely your client does not have RW permission to the pools

Zheng Yan wrote:

it's likely your client does not have RW permission to the pools

I don't think the problem is permission related:

- cephfs is mounted without specifying the user > client.admin with full permissins
accessing, writing and modifying files works fine on the NFS server and other cephfs clients with the same mount point

The problem only occurs on the desktop machines with re-exported cephfs over NFS.

it's likely the client on your desktop machine does not RW permission to the pools. please try doing direct write on the ceph-fuse mount point. (dd if=/dev/zero bs=4k count=1 of=test oflag=direct)

The desktop machine do not have access to the ceph network at all. That's why I have to use a NFS gateway.

dd on NFS server (ceph-fuse mount point):

/srv/ceph/adm/temp/test$ dd if=/dev/zero bs=4k count=1 of=test oflag=direct
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000483719 s, 8.5 MB/s

Editing the file with vi is OK on the server

dd on NFS client (NFS mount point of re-exported ceph-fuse mount):

:/ceph/adm/temp/test$ dd if=/dev/zero bs=4k count=1 of=test oflag=direct
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.151032 s, 27.1 kB/s

vi fails again and creates two swap files:

:/ceph/adm/temp/test$ ls al
total 5
drwxr-xr-x 1 blinke cb 4096 Jul 2 09:55 .
drwxrwxrwt 1 blinke support 20732144 Jun 30 16:45 ..
-rw-r--r- 1 blinke cb 4096 Jul 2 09:54 test
---------- 1 blinke cb 0 Feb 16 1974 .test.swo
---------- 1 blinke cb 0 Feb 16 1974 .test.swp

The date of the swap files have changed compared to the test I've done yesterday.

no idea what happened, please try using newer kernel (such as 4.0 kernel) on both NFS server and NFS client

I was able to update the kernel on the NFS server to version 4.1.4 today, which also allows me to use the kernel client instead of ceph-fuse.

The issue is resolved by the update, accessing the files works without problems or spurious files now.