Actions
Bug #56446
openTest failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
Testing
Target version:
% Done:
0%
Source:
Community (dev)
Tags:
backport_processed
Backport:
reef,quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Failure is while writing out data to the file system.
2022-06-30T12:14:29.372 INFO:tasks.cephfs_test_runner:test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) ... ERROR
2022-06-30T12:14:29.373 INFO:tasks.cephfs_test_runner:
2022-06-30T12:14:29.373 INFO:tasks.cephfs_test_runner:======================================================================
2022-06-30T12:14:29.373 INFO:tasks.cephfs_test_runner:ERROR: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2022-06-30T12:14:29.374 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-06-30T12:14:29.375 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-06-30T12:14:29.375 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/cephfs/test_client_limits.py", line 305, in test_client_cache_size
2022-06-30T12:14:29.376 INFO:tasks.cephfs_test_runner: self._test_client_cache_size(True)
2022-06-30T12:14:29.376 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/cephfs/test_client_limits.py", line 280, in _test_client_cache_size
2022-06-30T12:14:29.377 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(mkdir_script.format(path=dir_path, num_dirs=num_dirs))
2022-06-30T12:14:29.377 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/cephfs/mount.py", line 705, in run_python
2022-06-30T12:14:29.377 INFO:tasks.cephfs_test_runner: p.wait()
2022-06-30T12:14:29.378 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 161, in wait
2022-06-30T12:14:29.378 INFO:tasks.cephfs_test_runner: self._raise_for_status()
2022-06-30T12:14:29.378 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 183, in _raise_for_status
2022-06-30T12:14:29.379 INFO:tasks.cephfs_test_runner: node=self.hostname, label=self.label
2022-06-30T12:14:29.379 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'adjust-ulimits daemon-helper kill python3 -c \'\nimport os\nos.mkdir("/home/ubuntu/cephtest/mnt.0/testdir")\nfor n in range(0, 1000
):\n os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir/dir{0}".format(n))\n\''
2022-06-30T12:14:29.380 INFO:tasks.cephfs_test_runner:
Doesn't look like something that would have happened due to the PRs being tested or an issue with the client/mds. Seems more of a su/sudo thing in teuthology (we merged some changes to those recently).
Updated by Rishabh Dave almost 2 years ago
Path directory /home/ubuntu/cephtest/mnt.0/testdir is created twice. Copying following from https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6905028/teuthology.log -
2022-06-30T12:14:02.216 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits daemon-helper kill python3 -c '
2022-06-30T12:14:02.217 DEBUG:teuthology.orchestra.run.smithi032:> import os
2022-06-30T12:14:02.218 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir")
2022-06-30T12:14:02.218 DEBUG:teuthology.orchestra.run.smithi032:> for n in range(0, 1000):
2022-06-30T12:14:02.219 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir/dir{0}".format(n))
2022-06-30T12:14:02.219 DEBUG:teuthology.orchestra.run.smithi032:> '
2022-06-30T12:14:16.446 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits daemon-helper kill python3 -c '
2022-06-30T12:14:16.446 DEBUG:teuthology.orchestra.run.smithi032:> import os
2022-06-30T12:14:16.447 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir")
2022-06-30T12:14:16.447 DEBUG:teuthology.orchestra.run.smithi032:> for n in range(0, 1000):
2022-06-30T12:14:16.447 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir/dir{0}".format(n))
2022-06-30T12:14:16.448 DEBUG:teuthology.orchestra.run.smithi032:> '
2022-06-30T12:14:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:Traceback (most recent call last):
2022-06-30T12:14:16.589 INFO:teuthology.orchestra.run.smithi032.stderr: File "<string>", line 3, in <module>
2022-06-30T12:14:16.590 INFO:teuthology.orchestra.run.smithi032.stderr:FileExistsError: [Errno 17] File exists: '/home/ubuntu/cephtest/mnt.0/testdir'
2022-06-30T12:14:16.762 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command failed with exit status 1
2022-06-30T12:14:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1
So a better cleanup should fix this issue. However this can't be the root cause since the test is pretty old and bug was caught only recently. Some other changes in QA must've triggered this.
Updated by Rishabh Dave almost 2 years ago
- Status changed from New to In Progress
Updated by Venky Shankar almost 2 years ago
Rishabh Dave wrote:
Path directory
/home/ubuntu/cephtest/mnt.0/testdiris created twice. Copying following from https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6905028/teuthology.log -[...]
[...]
So a better cleanup should fix this issue. However this can't be the root cause since the test is pretty old and bug was caught only recently. Some other changes in QA must've triggered this.
Yeh. Running the test twice doesn't explain why we are seeing this failure recently.
Updated by Venky Shankar over 1 year ago
Rishabh - I'm taking this one since its blocking testing for https://tracker.ceph.com/issues/57985
Updated by Venky Shankar over 1 year ago
- Assignee changed from Rishabh Dave to Venky Shankar
Updated by Venky Shankar over 1 year ago
- Status changed from In Progress to Fix Under Review
- Pull request ID set to 49912
Updated by Venky Shankar about 1 year ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot about 1 year ago
- Copied to Backport #59030: quincy: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) added
Updated by Backport Bot about 1 year ago
- Copied to Backport #59031: pacific: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) added
Updated by Backport Bot about 1 year ago
- Copied to Backport #59032: pacific: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) added
Updated by Backport Bot about 1 year ago
- Copied to Backport #59033: quincy: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) added
Updated by Venky Shankar about 1 year ago
- Tags deleted (
backport_processed) - Backport changed from quincy, pacific to reef,quincy,pacific
Updated by Backport Bot about 1 year ago
- Copied to Backport #59430: reef: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) added
Updated by Kotresh Hiremath Ravishankar 12 months ago
reef - https://pulpito.ceph.com/yuriw-2023-05-15_15:22:39-fs-wip-yuri6-testing-2023-04-26-1247-reef-distro-default-smithi/7274310
reef backport is yet to be merged. Putting this for record.
Updated by Kotresh Hiremath Ravishankar 12 months ago
Actions