Ceph » CephFS

15 minutes is unnecessarily large as a default value for timeout for a command. Not having to wait unnecessarily on a crash of a command will reduce teuthology's test scheduling queue and will save individual developer's time while running tests locally.

In following vstart_runner log snippet execution was evidently stuck waiting for "ls subdir" for several minutes -

2022-02-09 15:10:02,134.134 DEBUG:__main__:> stat --file-system --printf=%T
 -- /tmp/tmpyzflq5ka/mnt.0
2022-02-09 15:10:02,141.141 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /tmp/tmpyzflq5ka/mnt.0
2022-02-09 15:10:02,141.141 DEBUG:__main__:> sudo chmod 1777 /tmp/tmpyzflq5ka/mnt.0
2022-02-09 15:10:02,163.163 DEBUG:__main__:> ls -l subdir/
^C2022-02-09 15:13:01,118.118 INFO:__main__:Stopped test: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan)
That when injecting a dentry into a fragmented directory, we put it in the right fragment. in 221.112574s

<<<<Traceback deleted>>>>

KeyboardInterrupt
2022-02-09 15:13:01,158.158 INFO:__main__:test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan)
2022-02-09 15:13:01,158.158 INFO:__main__:That when injecting a dentry into a fragmented directory, we put it in the right fragment. ...
^C

Next execution was fine -

2022-02-09 15:17:18,871.871 DEBUG:__main__:> stat --file-system --printf=%T
 -- /tmp/tmpkdlweyce/mnt.0
2022-02-09 15:17:18,877.877 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /tmp/tmpkdlweyce/mnt.0
2022-02-09 15:17:18,877.877 DEBUG:__main__:> sudo chmod 1777 /tmp/tmpkdlweyce/mnt.0
2022-02-09 15:17:18,877.877 DEBUG:__main__:> sudo chmod 1777 /tmp/tmpkdlweyce/mnt.0
2022-02-09 15:17:18,894.894 DEBUG:__main__:> ls -l subdir/
2022-02-09 15:17:19,066.066 DEBUG:__main__:> sudo bash -c 'cat subdir/21'

Another occurrence of the same issue -

2022-02-09 15:24:27,481.481 DEBUG:__main__:> ./bin/ceph --admin-daemon /tmp/ceph-asok.AfnaLh/client.0.35623.asok status
2022-02-09 15:24:27,660.660 DEBUG:__main__:> stat --file-system --printf=%T
 -- /tmp/tmp0bx1pf0b/mnt.0
2022-02-09 15:24:27,666.666 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /tmp/tmp0bx1pf0b/mnt.0
2022-02-09 15:24:27,667.667 DEBUG:__main__:> sudo chmod 1777 /tmp/tmp0bx1pf0b/mnt.0
2022-02-09 15:24:27,686.686 DEBUG:__main__:> ls -l subdir/
^C2022-02-09 16:11:28,206.206 INFO:__main__:Stopped test: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan)
That when injecting a dentry into a fragmented directory, we put it in the right fragment. in 2854.454991s

<<<<traceback deleted>>>>

KeyboardInterrupt
2022-02-09 16:11:28,319.319 INFO:__main__:test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan)
2022-02-09 16:11:28,320.320 INFO:__main__:That when injecting a dentry into a fragmented directory, we put it in the right fragment. ...
^C

Test again ran successfully next time -

2022-02-09 16:20:37,327.327 DEBUG:__main__:> ./bin/ceph --admin-daemon /tmp/ceph-asok.AfnaLh/client.0.75616.asok status
2022-02-09 16:20:37,683.683 DEBUG:__main__:> stat --file-system --printf=%T
 -- /tmp/tmpt1q7uunh/mnt.0
2022-02-09 16:20:37,707.707 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /tmp/tmpt1q7uunh/mnt.0
2022-02-09 16:20:37,708.708 DEBUG:__main__:> sudo chmod 1777 /tmp/tmpt1q7uunh/mnt.0
2022-02-09 16:20:37,741.741 DEBUG:__main__:> ls -l subdir/
2022-02-09 16:20:39,095.095 DEBUG:__main__:> sudo bash -c 'cat subdir/21'
2022-02-09 16:20:39,162.162 DEBUG:__main__:> ./bin/ceph fs dump --format=json