Ceph » CephFS

The DelegTimeout test case itself it's buggy:

299 TEST(LibCephFS, DelegTimeout) {
300   struct ceph_mount_info *cmount;
301   ASSERT_EQ(ceph_create(&cmount, NULL), 0);
302   ASSERT_EQ(ceph_conf_read_file(cmount, NULL), 0);
303   ASSERT_EQ(0, ceph_conf_parse_env(cmount, NULL)); 
304   ASSERT_EQ(ceph_mount(cmount, "/"), 0);
305   // tweak timeout to run quickly, since we don't plan to return it anyway
306   ASSERT_EQ(ceph_set_deleg_timeout(cmount, 2), 0);
307    
308   Inode *root, *file;
309   ASSERT_EQ(ceph_ll_lookup_root(cmount, &root), 0);
310    
311   char filename[32];
312   sprintf(filename, "delegtimeo%x", getpid());
313 
314   Fh *fh;
315   struct ceph_statx stx;
316   UserPerm *perms = ceph_mount_perms(cmount);
317 
318   ASSERT_EQ(ceph_ll_create(cmount, root, filename, 0666,
319                     O_RDWR|O_CREAT|O_EXCL, &file, &fh, &stx, 0, 0, perms), 0); 
320 
321   /* Reopen read-only */
322   ASSERT_EQ(ceph_ll_close(cmount, fh), 0);
323   ASSERT_EQ(ceph_ll_open(cmount, file, O_RDONLY, &fh, perms), 0);
324 
325   std::atomic_bool recalled(false);
326   ASSERT_EQ(ceph_ll_delegation_wait(cmount, fh, CEPH_DELEGATION_RD, dummy_deleg_cb, &recalled), 0);
327   std::atomic_bool opened(false);
328   std::thread breaker1(open_breaker_func, nullptr, filename, O_RDWR, &opened);
329   breaker1.join();
330   ASSERT_EQ(recalled.load(), true);
331   ASSERT_EQ(ceph_ll_getattr(cmount, root, &stx, 0, 0, perms), -CEPHFS_ENOTCONN);
332   ceph_release(cmount);
333 }

Because the MDS Locker allow multiple clients to open the file with rw and the lock state will be:

[LOCK_MIX]       = { 0,         false, LOCK_MIX,  0,    0,   REQ, ANY, 0,   0,   0,   CEPH_CAP_GRD|CEPH_CAP_GWR|CEPH_CAP_GLAZYIO,0,0,CEPH_CAP_GRD },

So the second mounter in child open_breaker_func thread when opening the file with rw mode, the client won't break the delegation in main thread and the child thread could finish very fast in 2 seconds, which is from 306 ASSERT_EQ(ceph_set_deleg_timeout(cmount, 2), 0); in main thread.

That means when in main thread Line#331, the delegation hasn't timedout yet. When the delegation times out it will umount the mounter directly and then when accessing the mounter later it should return -CEPHFS_ENOTCONN.

main:
https://pulpito.ceph.com/mchangir-2024-03-22_09:48:09-fs:libcephfs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616279/

2024-03-22T10:37:17.172 INFO:tasks.workunit.client.0.smithi077.stdout:[  FAILED  ] TestClient.AlternateNameSymlink
2024-03-22T10:37:17.172 INFO:tasks.workunit.client.0.smithi077.stdout:[  FAILED  ] TestClient.AlternateNameLink

https://pulpito.ceph.com/mchangir-2024-03-22_09:48:09-fs:libcephfs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616281/

2024-03-22T10:42:16.714 INFO:tasks.workunit.client.0.smithi089.stdout:[  FAILED  ] LibCephFS.LayoutVerifyDefaultLayout
2024-03-22T10:42:16.714 INFO:tasks.workunit.client.0.smithi089.stdout:[  FAILED  ] LibCephFS.LayoutSetAndVerifyNewAndInheritedLayout
2024-03-22T10:42:16.714 INFO:tasks.workunit.client.0.smithi089.stdout:[  FAILED  ] LibCephFS.GetAndSetDirDistribution
2024-03-22T10:42:16.714 INFO:tasks.workunit.client.0.smithi089.stdout:[  FAILED  ] LibCephFS.GetAndSetDirRandom

Milind Changire wrote:

main:
https://pulpito.ceph.com/mchangir-2024-03-22_09:48:09-fs:libcephfs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616279/
[...]

https://pulpito.ceph.com/mchangir-2024-03-22_09:48:09-fs:libcephfs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616281/
[...]

Hi Milind,

This should be a different issue, not failed from [ FAILED ] LibCephFS.DelegTimeout (75 ms).