Project

General

Profile

Bug #56446

Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)

Added by Venky Shankar over 1 year ago. Updated 10 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
Testing
Target version:
% Done:

0%

Source:
Community (dev)
Tags:
backport_processed
Backport:
reef,quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Seen here: https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6905028/

Failure is while writing out data to the file system.

2022-06-30T12:14:29.372 INFO:tasks.cephfs_test_runner:test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) ... ERROR
2022-06-30T12:14:29.373 INFO:tasks.cephfs_test_runner:
2022-06-30T12:14:29.373 INFO:tasks.cephfs_test_runner:======================================================================
2022-06-30T12:14:29.373 INFO:tasks.cephfs_test_runner:ERROR: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2022-06-30T12:14:29.374 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-06-30T12:14:29.375 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-06-30T12:14:29.375 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/cephfs/test_client_limits.py", line 305, in test_client_cache_size
2022-06-30T12:14:29.376 INFO:tasks.cephfs_test_runner:    self._test_client_cache_size(True)
2022-06-30T12:14:29.376 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/cephfs/test_client_limits.py", line 280, in _test_client_cache_size
2022-06-30T12:14:29.377 INFO:tasks.cephfs_test_runner:    self.mount_a.run_python(mkdir_script.format(path=dir_path, num_dirs=num_dirs))
2022-06-30T12:14:29.377 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_5292901bc999daa0d7ac128764a3f27c4233c252/qa/tasks/cephfs/mount.py", line 705, in run_python
2022-06-30T12:14:29.377 INFO:tasks.cephfs_test_runner:    p.wait()
2022-06-30T12:14:29.378 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 161, in wait
2022-06-30T12:14:29.378 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2022-06-30T12:14:29.378 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 183, in _raise_for_status
2022-06-30T12:14:29.379 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2022-06-30T12:14:29.379 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'adjust-ulimits daemon-helper kill python3 -c \'\nimport os\nos.mkdir("/home/ubuntu/cephtest/mnt.0/testdir")\nfor n in range(0, 1000
):\n    os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir/dir{0}".format(n))\n\''
2022-06-30T12:14:29.380 INFO:tasks.cephfs_test_runner:

Doesn't look like something that would have happened due to the PRs being tested or an issue with the client/mds. Seems more of a su/sudo thing in teuthology (we merged some changes to those recently).


Related issues

Copied to CephFS - Backport #59030: quincy: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) In Progress
Copied to CephFS - Backport #59031: pacific: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) Duplicate
Copied to CephFS - Backport #59032: pacific: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) Resolved
Copied to CephFS - Backport #59033: quincy: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) Duplicate
Copied to CephFS - Backport #59430: reef: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) Resolved

History

#1 Updated by Rishabh Dave over 1 year ago

  • Assignee set to Rishabh Dave

#2 Updated by Rishabh Dave over 1 year ago

Path directory /home/ubuntu/cephtest/mnt.0/testdir is created twice. Copying following from https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6905028/teuthology.log -

2022-06-30T12:14:02.216 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits daemon-helper kill python3 -c '
2022-06-30T12:14:02.217 DEBUG:teuthology.orchestra.run.smithi032:> import os
2022-06-30T12:14:02.218 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir")
2022-06-30T12:14:02.218 DEBUG:teuthology.orchestra.run.smithi032:> for n in range(0, 1000):
2022-06-30T12:14:02.219 DEBUG:teuthology.orchestra.run.smithi032:>     os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir/dir{0}".format(n))
2022-06-30T12:14:02.219 DEBUG:teuthology.orchestra.run.smithi032:> '
2022-06-30T12:14:16.446 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits daemon-helper kill python3 -c '
2022-06-30T12:14:16.446 DEBUG:teuthology.orchestra.run.smithi032:> import os
2022-06-30T12:14:16.447 DEBUG:teuthology.orchestra.run.smithi032:> os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir")
2022-06-30T12:14:16.447 DEBUG:teuthology.orchestra.run.smithi032:> for n in range(0, 1000):
2022-06-30T12:14:16.447 DEBUG:teuthology.orchestra.run.smithi032:>     os.mkdir("/home/ubuntu/cephtest/mnt.0/testdir/dir{0}".format(n))
2022-06-30T12:14:16.448 DEBUG:teuthology.orchestra.run.smithi032:> '
2022-06-30T12:14:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:Traceback (most recent call last):
2022-06-30T12:14:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:  File "<string>", line 3, in <module>
2022-06-30T12:14:16.590 INFO:teuthology.orchestra.run.smithi032.stderr:FileExistsError: [Errno 17] File exists: '/home/ubuntu/cephtest/mnt.0/testdir'
2022-06-30T12:14:16.762 INFO:teuthology.orchestra.run.smithi032.stderr:daemon-helper: command failed with exit status 1
2022-06-30T12:14:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1

So a better cleanup should fix this issue. However this can't be the root cause since the test is pretty old and bug was caught only recently. Some other changes in QA must've triggered this.

#3 Updated by Rishabh Dave over 1 year ago

  • Status changed from New to In Progress

#4 Updated by Venky Shankar over 1 year ago

Rishabh Dave wrote:

Path directory /home/ubuntu/cephtest/mnt.0/testdir is created twice. Copying following from https://pulpito.ceph.com/vshankar-2022-06-29_09:19:00-fs-wip-vshankar-testing-20220627-100931-testing-default-smithi/6905028/teuthology.log -

[...]

[...]

So a better cleanup should fix this issue. However this can't be the root cause since the test is pretty old and bug was caught only recently. Some other changes in QA must've triggered this.

Yeh. Running the test twice doesn't explain why we are seeing this failure recently.

#5 Updated by Venky Shankar over 1 year ago

Rishabh, did you get to RCA this?

#6 Updated by Venky Shankar about 1 year ago

Rishabh - I'm taking this one since its blocking testing for https://tracker.ceph.com/issues/57985

#7 Updated by Venky Shankar about 1 year ago

  • Assignee changed from Rishabh Dave to Venky Shankar

#8 Updated by Venky Shankar about 1 year ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 49912

#9 Updated by Venky Shankar about 1 year ago

  • Status changed from Fix Under Review to Pending Backport

#10 Updated by Backport Bot about 1 year ago

  • Copied to Backport #59030: quincy: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) added

#11 Updated by Backport Bot about 1 year ago

  • Copied to Backport #59031: pacific: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) added

#12 Updated by Backport Bot about 1 year ago

  • Copied to Backport #59032: pacific: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) added

#13 Updated by Backport Bot about 1 year ago

  • Tags set to backport_processed

#14 Updated by Backport Bot about 1 year ago

  • Copied to Backport #59033: quincy: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) added

#15 Updated by Venky Shankar 12 months ago

  • Tags deleted (backport_processed)
  • Backport changed from quincy, pacific to reef,quincy,pacific

#16 Updated by Backport Bot 12 months ago

  • Copied to Backport #59430: reef: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) added

#17 Updated by Backport Bot 12 months ago

  • Tags set to backport_processed

Also available in: Atom PDF