test_disk_quota_exceeeded_error from src/test/pybind/test_cephfs.py fails on this teuthology run - http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/7193845
2023-03-04T09:56:26.878 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_disk_quota_exceeeded_error ... FAIL
2023-03-04T09:56:26.891 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_empty_snapshot_info ... ERROR
2023-03-04T09:56:26.896 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_snapshot_info ... ERROR
2023-03-04T09:56:26.899 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_set_mount_timeout_post_mount ... ERROR
2023-03-04T09:56:26.902 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_set_mount_timeout ... ERROR
2023-03-04T09:56:26.971 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_set_mount_timeout_lt0 ... ERROR
2023-03-04T09:56:26.972 INFO:tasks.workunit.client.0.smithi114.stderr:
Traceback for test_disk_quota_exceeeded_error -
2023-03-04T09:56:26.983 INFO:tasks.workunit.client.0.smithi114.stderr:======================================================================
2023-03-04T09:56:26.984 INFO:tasks.workunit.client.0.smithi114.stderr:FAIL: test_cephfs.test_disk_quota_exceeeded_error
2023-03-04T09:56:26.984 INFO:tasks.workunit.client.0.smithi114.stderr:----------------------------------------------------------------------
2023-03-04T09:56:26.984 INFO:tasks.workunit.client.0.smithi114.stderr:Traceback (most recent call last):
2023-03-04T09:56:26.984 INFO:tasks.workunit.client.0.smithi114.stderr: File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
2023-03-04T09:56:26.985 INFO:tasks.workunit.client.0.smithi114.stderr: self.test(*self.arg)
2023-03-04T09:56:26.985 INFO:tasks.workunit.client.0.smithi114.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_cephfs.py", line 859, in test_disk_quota_exceeeded_error
2023-03-04T09:56:26.985 INFO:tasks.workunit.client.0.smithi114.stderr: assert_raises(libcephfs.DiskQuotaExceeded, cephfs.write, fd, b"abcdeghiklmnopqrstuvwxyz1234567890qwertyuioddd", -1)
Every test that failed after test_disk_quota_exceeeded_error has same traceback -
2023-03-04T09:56:26.972 INFO:tasks.workunit.client.0.smithi114.stderr:ERROR: test_cephfs.test_empty_snapshot_info
2023-03-04T09:56:26.972 INFO:tasks.workunit.client.0.smithi114.stderr:----------------------------------------------------------------------
2023-03-04T09:56:26.973 INFO:tasks.workunit.client.0.smithi114.stderr:Traceback (most recent call last):
2023-03-04T09:56:26.973 INFO:tasks.workunit.client.0.smithi114.stderr: File "/usr/lib/python3/dist-packages/nose/case.py", line 267, in setUp
2023-03-04T09:56:26.973 INFO:tasks.workunit.client.0.smithi114.stderr: try_run(self.test, names)
2023-03-04T09:56:26.973 INFO:tasks.workunit.client.0.smithi114.stderr: File "/usr/lib/python3/dist-packages/nose/util.py", line 471, in try_run
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr: return func()
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_cephfs.py", line 29, in setup_test
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr: cephfs.rmdir(b"/" + dent.d_name)
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr: File "cephfs.pyx", line 1276, in cephfs.LibCephFS.rmdir
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr:cephfs.ObjectNotEmpty: error in rmdir /dir-1: Directory not empty [Errno 39]
setUp() hits error on running rmdir /dir-1. Normally this doesn't occur which means this dir-1 is normally empty. This implies unclean cleanup/teardown in previous test which is test_disk_quota_exceeeded_error.
Code of test_disk_quota_exceeeded_error -
@with_setup(setup_test)
def test_disk_quota_exceeeded_error():
cephfs.mkdir("/dir-1", 0o755)
cephfs.setxattr("/dir-1", "ceph.quota.max_bytes", b"5", 0)
fd = cephfs.open(b'/dir-1/file-1', 'w', 0o755)
assert_raises(libcephfs.DiskQuotaExceeded, cephfs.write, fd, b"abcdeghiklmnopqrstuvwxyz", 0)
cephfs.close(fd)
cephfs.unlink(b"/dir-1/file-1")
Specifically following code from above test is quota related code where failure occurs -
cephfs.setxattr("/dir-1", "ceph.quota.max_bytes", b"5", 0)
fd = cephfs.open(b'/dir-1/file-1', 'w', 0o755)
assert_raises(libcephfs.DiskQuotaExceeded, cephfs.write, fd, b"abcdeghiklmnopqrstuvwxyz", 0)
And the last line of this test method (which is cephfs.unlink(b"/dir-1/file-1") is not executed. Thus, dir-1 is not empty and setUp() fails on attempting to run rmdir /dir-1.
In short the cause for this bug is same as that of this tracker ticket.