Ceph » CephFS

A quick look at the logs shows that there are 4 clients running this test simultaneously. I wonder if this something that used to succeed before. Because these clients seem to be interfering with each other, setting and removing quotas.

If that's the case, a possible fix would be to have each client create it's own test directory. Something like the diff below:

diff --git a/qa/workunits/fs/quota/quota.sh b/qa/workunits/fs/quota/quota.sh
index 1315be6d8609..d6e59317ecdf 100755
--- a/qa/workunits/fs/quota/quota.sh
+++ b/qa/workunits/fs/quota/quota.sh
@@ -25,8 +25,9 @@ function write_file()
        return 0
 }

-mkdir quota-test
-cd quota-test
+testdir=`hostname -A`-quota-test
+mkdir $testdir
+cd $testdir

 # bytes
 setfattr . -n ceph.quota.max_bytes -v 100000000  # 100m
@@ -123,6 +124,6 @@ expect_false setfattr -n ceph.quota -v "max_bytes=-1 max_files=-1" .
 #addme

 cd ..
-rm -rf quota-test
+rm -rf $testdir

 echo OK

Patrick Donnelly wrote:

Luis Henriques wrote:

A quick look at the logs shows that there are 4 clients running this test simultaneously. I wonder if this something that used to succeed before. Because these clients seem to be interfering with each other, setting and removing quotas.

If that's the case, a possible fix would be to have each client create it's own test directory. Something like the diff below:

[...]

Luis, I took a look. Each client gets its own subdirectory in the CephFS mount:

[...]

Emphasis on "mkdir p - /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd -- /home/ubuntu/cephtest/mnt.2/client.2/tmp". mnt.2 is the CephFS root.

Mind taking another look?

Ah, sorry! I missed that. Sure, I'll have another look at the logs.

Luis Henriques wrote:

Patrick Donnelly wrote:

Luis Henriques wrote:

A quick look at the logs shows that there are 4 clients running this test simultaneously. I wonder if this something that used to succeed before. Because these clients seem to be interfering with each other, setting and removing quotas.

If that's the case, a possible fix would be to have each client create it's own test directory. Something like the diff below:

[...]

Luis, I took a look. Each client gets its own subdirectory in the CephFS mount:

[...]

Emphasis on "mkdir p - /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd -- /home/ubuntu/cephtest/mnt.2/client.2/tmp". mnt.2 is the CephFS root.

Mind taking another look?

Ah, sorry! I missed that. Sure, I'll have another look at the logs.

Great, thanks for having a look! I'll assign this to you for now :)

Quick update: Looking further at the logs helped me... getting more confused :-)

So, all the 4 clients are failing when they try to create the 'shouldbefail' file, because that operation should fail due to max_files quota being set.

What I'm seeing is that both client.0 and client.1 call insert_dentry_inode() for the 'quota-test' dir several times, with different vino. We have 4 clients: clients 2 and 3 call this function only once (with a unique vino each); the other 2 clients call it 4 times, with the vinos for all the 4 clients:

• client.2: 0x100000007d4
• client.3: 0x10000000bbd
• client.0: 0x10000000002, 0x100000003eb, 0x100000007d4, 0x10000000bbd
• client.1: 0x100000003eb, 0x10000000002, 0x100000007d4, 0x10000000bbd

This looks wrong to me, each client shouldn't be reading inside each others client.$ID where the 'quota-test' dirs are.

So, my current theory is that the MDS is doing something wrong setting the snaprealms and the clients are receiving the wrong snaprealms for their quota-test dirs. I'll try dig a bit deeper into the MDS code and see if I can find something, but maybe the above description rings a bell to anyone (Yan? :-) )

And another update: I can not understand why there are two clients (both on smithi071 btw) that do a readdir in the root directory. See for ex. aroung 03:08:36.514 in client.0.

Initially I thought this could also be a prob with casting/truncation because the readdir was being done on 0x1 instead of 0x10000000001 (I even thought about 32bits clients, but they all seem to be 64 bits). But then the client.1 should be readdir'ing 0x100000003ea, and still doing it on 0x1.

Patrick Donnelly wrote:

/ceph/teuthology-archive/pdonnell-2020-11-04_17:39:34-fs-wip-pdonnell-testing-20201103.210407-distro-basic-smithi/5590496/teuthology.log
/ceph/teuthology-archive/pdonnell-2020-11-04_17:39:34-fs-wip-pdonnell-testing-20201103.210407-distro-basic-smithi/5590512/teuthology.log

(might be a different failure.)

Definitely a different failure, but this time I think it's easy to understand what's going on. It's a regression introduced (by me!) with the fix for cross-quota-tree renames.

Looks like the issue started happening in kernel 5.8 with commit dffdcd71458e ("ceph: allow rename operation under different quota realms"). My first guess is that the MDSs aren't updating inodes RSTATs immediately after a truncate operation:

+ mkdir files limit
+ truncate files/file -s 10G
+ setfattr limit -n ceph.quota.max_bytes -v 1000000
+ expect_false mv files limit/
+ set -x
+ mv files limit/
+ return 1

Is there a reason for the MDSs to postpone this update? I'll need to check what the fuse client is doing here (not even sure the fuse client allows renames across quota realms).

The kernel client code is optimized to buffer the new file size when doing the truncate syscall:

        if (ia_valid & ATTR_SIZE) {
                if ((issued & CEPH_CAP_FILE_EXCL) &&
                    attr->ia_size > inode->i_size) {
                ...
                }
                } else if ((issued & CEPH_CAP_FILE_SHARED) == 0 ||
                           attr->ia_size != inode->i_size) {
                ...
                }
         }

A trivial fix would be to always send it, even if we have Fx caps. This would have a performance impact, of course but I believe this is what the fuse client is doing.

Thanks for checking Luis, I made a new ticket here: https://tracker.ceph.com/issues/48203

Let's move the discussion over there.