Bug #56446: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) - CephFS - Ceph

Bug #56446

Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)

Added by Venky Shankar about 1 year ago. Updated 4 months ago.

Status:

Pending Backport

Priority:

Normal

Assignee:

Venky Shankar

Category:

Testing

Target version:

Ceph - v18.0.0

% Done:

Source:

Community (dev)

Tags:

backport_processed

Backport:

reef,quincy,pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

Labels (FS):

qa, qa-failure

Pull request ID:

49912

Crash signature (v1):

Crash signature (v2):

Description

Seen here: https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6905028/

Failure is while writing out data to the file system.

2022-06-30T12:14:29.372 INFO:tasks.cephfs_test_runner:test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) ... ERROR
2022-06-30T12:14:29.373 INFO:tasks.cephfs_test_runner:
2022-06-30T12:14:29.373 INFO:tasks.cephfs_test_runner:======================================================================
2022-06-30T12:14:29.373 INFO:tasks.cephfs_test_runner:ERROR: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2022-06-30T12:14:29.374 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-06-30T12:14:29.375 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-06-30T12:14:29.375 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/cephfs/test_client_limits.py", line 305, in test_client_cache_size
2022-06-30T12:14:29.376 INFO:tasks.cephfs_test_runner:    self._test_client_cache_size(True)
2022-06-30T12:14:29.376 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/cephfs/test_client_limits.py", line 280, in _test_client_cache_size
2022-06-30T12:14:29.377 INFO:tasks.cephfs_test_runner:    self.mount_a.run_python(mkdir_script.format(path=dir_path, num_dirs=num_dirs))
2022-06-30T12:14:29.377 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/cephfs/mount.py", line 705, in run_python
2022-06-30T12:14:29.377 INFO:tasks.cephfs_test_runner:    p.wait()
2022-06-30T12:14:29.378 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 161, in wait
2022-06-30T12:14:29.378 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2022-06-30T12:14:29.378 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 183, in _raise_for_status
2022-06-30T12:14:29.379 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2022-06-30T12:14:29.379 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'adjust-ulimits daemon-helper kill python3 -c \'\nimport os\nos.mkdir("/home/ubuntu/cephtest/mnt.0/testdir")\nfor n in range(0, 1000
):\n    os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir/dir{0}".format(n))\n\''
2022-06-30T12:14:29.380 INFO:tasks.cephfs_test_runner:

Doesn't look like something that would have happened due to the PRs being tested or an issue with the client/mds. Seems more of a su/sudo thing in teuthology (we merged some changes to those recently).

Related issues

	Copied to CephFS - Backport #59030: quincy: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)	In Progress
	Copied to CephFS - Backport #59031: pacific: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)	Duplicate
	Copied to CephFS - Backport #59032: pacific: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)	Resolved
	Copied to CephFS - Backport #59033: quincy: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)	Duplicate
	Copied to CephFS - Backport #59430: reef: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)	Resolved

History

#1 Updated by Rishabh Dave about 1 year ago

Assignee set to Rishabh Dave

#2 Updated by Rishabh Dave about 1 year ago

Path directory /home/ubuntu/cephtest/mnt.0/testdir is created twice. Copying following from https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6905028/teuthology.log -

2022-06-30T12:14:02.216 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits daemon-helper kill python3 -c '
2022-06-30T12:14:02.217 DEBUG:teuthology.orchestra.run.smithi032:> import os
2022-06-30T12:14:02.218 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir")
2022-06-30T12:14:02.218 DEBUG:teuthology.orchestra.run.smithi032:> for n in range(0, 1000):
2022-06-30T12:14:02.219 DEBUG:teuthology.orchestra.run.smithi032:>     os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir/dir{0}".format(n))
2022-06-30T12:14:02.219 DEBUG:teuthology.orchestra.run.smithi032:> '

2022-06-30T12:14:16.446 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits daemon-helper kill python3 -c '
2022-06-30T12:14:16.446 DEBUG:teuthology.orchestra.run.smithi032:> import os
2022-06-30T12:14:16.447 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir")
2022-06-30T12:14:16.447 DEBUG:teuthology.orchestra.run.smithi032:> for n in range(0, 1000):
2022-06-30T12:14:16.447 DEBUG:teuthology.orchestra.run.smithi032:>     os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir/dir{0}".format(n))
2022-06-30T12:14:16.448 DEBUG:teuthology.orchestra.run.smithi032:> '
2022-06-30T12:14:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:Traceback (most recent call last):
2022-06-30T12:14:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:  File "<string>", line 3, in <module>
2022-06-30T12:14:16.590 INFO:teuthology.orchestra.run.smithi032.stderr:FileExistsError: [Errno 17] File exists: '/home/ubuntu/cephtest/mnt.0/testdir'
2022-06-30T12:14:16.762 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command failed with exit status 1
2022-06-30T12:14:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1

So a better cleanup should fix this issue. However this can't be the root cause since the test is pretty old and bug was caught only recently. Some other changes in QA must've triggered this.

#3 Updated by Rishabh Dave about 1 year ago

Status changed from New to In Progress

#4 Updated by Venky Shankar about 1 year ago

Rishabh Dave wrote:

Path directory /home/ubuntu/cephtest/mnt.0/testdir is created twice. Copying following from https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6905028/teuthology.log -

[...]

[...]

So a better cleanup should fix this issue. However this can't be the root cause since the test is pretty old and bug was caught only recently. Some other changes in QA must've triggered this.

Yeh. Running the test twice doesn't explain why we are seeing this failure recently.