Project

General

Profile

Bug #56446

Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)

Added by Venky Shankar 5 months ago. Updated 5 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Testing
Target version:
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
quincy, pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Seen here: https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6905028/

Failure is while writing out data to the file system.

2022-06-30T12:14:29.372 INFO:tasks.cephfs_test_runner:test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) ... ERROR
2022-06-30T12:14:29.373 INFO:tasks.cephfs_test_runner:
2022-06-30T12:14:29.373 INFO:tasks.cephfs_test_runner:======================================================================
2022-06-30T12:14:29.373 INFO:tasks.cephfs_test_runner:ERROR: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2022-06-30T12:14:29.374 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-06-30T12:14:29.375 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-06-30T12:14:29.375 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/cephfs/test_client_limits.py", line 305, in test_client_cache_size
2022-06-30T12:14:29.376 INFO:tasks.cephfs_test_runner:    self._test_client_cache_size(True)
2022-06-30T12:14:29.376 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/cephfs/test_client_limits.py", line 280, in _test_client_cache_size
2022-06-30T12:14:29.377 INFO:tasks.cephfs_test_runner:    self.mount_a.run_python(mkdir_script.format(path=dir_path, num_dirs=num_dirs))
2022-06-30T12:14:29.377 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/cephfs/mount.py", line 705, in run_python
2022-06-30T12:14:29.377 INFO:tasks.cephfs_test_runner:    p.wait()
2022-06-30T12:14:29.378 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 161, in wait
2022-06-30T12:14:29.378 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2022-06-30T12:14:29.378 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 183, in _raise_for_status
2022-06-30T12:14:29.379 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2022-06-30T12:14:29.379 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'adjust-ulimits daemon-helper kill python3 -c \'\nimport os\nos.mkdir("/home/ubuntu/cephtest/mnt.0/testdir")\nfor n in range(0, 1000
):\n    os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir/dir{0}".format(n))\n\''
2022-06-30T12:14:29.380 INFO:tasks.cephfs_test_runner:

Doesn't look like something that would have happened due to the PRs being tested or an issue with the client/mds. Seems more of a su/sudo thing in teuthology (we merged some changes to those recently).

History

#1 Updated by Rishabh Dave 5 months ago

  • Assignee set to Rishabh Dave

#2 Updated by Rishabh Dave 5 months ago

Path directory /home/ubuntu/cephtest/mnt.0/testdir is created twice. Copying following from https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6905028/teuthology.log -

2022-06-30T12:14:02.216 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits daemon-helper kill python3 -c '
2022-06-30T12:14:02.217 DEBUG:teuthology.orchestra.run.smithi032:> import os
2022-06-30T12:14:02.218 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir")
2022-06-30T12:14:02.218 DEBUG:teuthology.orchestra.run.smithi032:> for n in range(0, 1000):
2022-06-30T12:14:02.219 DEBUG:teuthology.orchestra.run.smithi032:>     os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir/dir{0}".format(n))
2022-06-30T12:14:02.219 DEBUG:teuthology.orchestra.run.smithi032:> '
2022-06-30T12:14:16.446 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits daemon-helper kill python3 -c '
2022-06-30T12:14:16.446 DEBUG:teuthology.orchestra.run.smithi032:> import os
2022-06-30T12:14:16.447 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir")
2022-06-30T12:14:16.447 DEBUG:teuthology.orchestra.run.smithi032:> for n in range(0, 1000):
2022-06-30T12:14:16.447 DEBUG:teuthology.orchestra.run.smithi032:>     os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir/dir{0}".format(n))
2022-06-30T12:14:16.448 DEBUG:teuthology.orchestra.run.smithi032:> '
2022-06-30T12:14:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:Traceback (most recent call last):
2022-06-30T12:14:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:  File "<string>", line 3, in <module>
2022-06-30T12:14:16.590 INFO:teuthology.orchestra.run.smithi032.stderr:FileExistsError: [Errno 17] File exists: '/home/ubuntu/cephtest/mnt.0/testdir'
2022-06-30T12:14:16.762 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command failed with exit status 1
2022-06-30T12:14:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1

So a better cleanup should fix this issue. However this can't be the root cause since the test is pretty old and bug was caught only recently. Some other changes in QA must've triggered this.

#3 Updated by Rishabh Dave 5 months ago

  • Status changed from New to In Progress

#4 Updated by Venky Shankar 5 months ago

Rishabh Dave wrote:

Path directory /home/ubuntu/cephtest/mnt.0/testdir is created twice. Copying following from https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6905028/teuthology.log -

[...]

[...]

So a better cleanup should fix this issue. However this can't be the root cause since the test is pretty old and bug was caught only recently. Some other changes in QA must've triggered this.

Yeh. Running the test twice doesn't explain why we are seeing this failure recently.

#5 Updated by Venky Shankar 5 months ago

Rishabh, did you get to RCA this?

Also available in: Atom PDF