Project

General

Profile

Actions
Copy link

Bug #52887

closed

qa: Test failure: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable)

Added by Xiubo Li over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Xiubo Li
Category:
-
Target version:
Ceph - v17.0.0
% Done:

0%

Source:
Q/A
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
fs
Component(FS):
Labels (FS):
Pull request ID:
43767
Crash signature (v1):
Crash signature (v2):

Description

The teuthology test: https://pulpito.ceph.com/yuriw-2021-10-02_15:12:58-fs-wip-yuri2-testing-2021-10-01-0902-pacific-distro-basic-smithi/6417991/

2021-10-02T19:32:20.995 INFO:teuthology.orchestra.run.smithi037.stderr:2021-10-02T19:32:20.984+0000 7fe0d9636700  1 -- 172.21.15.37:0/3827556596 shutdown_connections
2021-10-02T19:32:20.995 INFO:teuthology.orchestra.run.smithi037.stderr:2021-10-02T19:32:20.984+0000 7fe0d9636700  1 -- 172.21.15.37:0/3827556596 wait complete.
2021-10-02T19:32:21.020 INFO:tasks.cephfs_test_runner:test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable) ... ERROR
2021-10-02T19:32:21.021 INFO:tasks.cephfs_test_runner:
2021-10-02T19:32:21.022 INFO:tasks.cephfs_test_runner:======================================================================
2021-10-02T19:32:21.022 INFO:tasks.cephfs_test_runner:ERROR: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable)
2021-10-02T19:32:21.022 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-10-02T19:32:21.023 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-10-02T19:32:21.023 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_6e2fe4ec19ea89ddb82f77e1df235925f6b0146e/qa/tasks/cephfs/test_openfiletable.py", line 76, in test_perf_counters
2021-10-02T19:32:21.023 INFO:tasks.cephfs_test_runner:    self.wait_until_true(lambda: self._check_oft_counter('omap_total_removes', 1), timeout=30)
2021-10-02T19:32:21.023 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_6e2fe4ec19ea89ddb82f77e1df235925f6b0146e/qa/tasks/ceph_test_case.py", line 206, in wait_until_true
2021-10-02T19:32:21.024 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2021-10-02T19:32:21.024 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries
2021-10-02T19:32:21.024 INFO:tasks.cephfs_test_runner:
2021-10-02T19:32:21.025 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-10-02T19:32:21.025 INFO:tasks.cephfs_test_runner:Ran 2 tests in 149.195s
2021-10-02T19:32:21.025 INFO:tasks.cephfs_test_runner:
2021-10-02T19:32:21.026 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2021-10-02T19:32:21.026 INFO:tasks.cephfs_test_runner:
2021-10-02T19:32:21.026 INFO:tasks.cephfs_test_runner:======================================================================
2021-10-02T19:32:21.027 INFO:tasks.cephfs_test_runner:ERROR: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable)
2021-10-02T19:32:21.027 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-10-02T19:32:21.027 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-10-02T19:32:21.027 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_6e2fe4ec19ea89ddb82f77e1df235925f6b0146e/qa/tasks/cephfs/test_openfiletable.py", line 76, in test_perf_counters
2021-10-02T19:32:21.028 INFO:tasks.cephfs_test_runner:    self.wait_until_true(lambda: self._check_oft_counter('omap_total_removes', 1), timeout=30)
2021-10-02T19:32:21.028 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_6e2fe4ec19ea89ddb82f77e1df235925f6b0146e/qa/tasks/ceph_test_case.py", line 206, in wait_until_true
2021-10-02T19:32:21.028 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2021-10-02T19:32:21.029 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries
2021-10-02T19:32:21.029 INFO:tasks.cephfs_test_runner:
2021-10-02T19:32:21.029 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/run_tasks.py", line 94, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_6e2fe4ec19ea89ddb82f77e1df235925f6b0146e/qa/tasks/cephfs_test_runner.py", line 211, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable)
2021-10-02T19:32:21.133 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e4e393f5685b475c9e54a5ff79008678
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/run_tasks.py", line 94, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_6e2fe4ec19ea89ddb82f77e1df235925f6b0146e/qa/tasks/cephfs_test_runner.py", line 211, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable)
2021-10-02T19:32:21.135 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner
2021-10-02T19:32:21.148 DEBUG:teuthology.run_tasks:Unwinding manager kclient
2021-10-02T19:32:21.193 INFO:tasks.kclient:Unmounting kernel clients...

Related issues 2 (0 open — 2 closed)

Has duplicate CephFS - Bug #52996: qa: test_perf_counters via test_openfiletableDuplicate

Actions
Copied to CephFS - Backport #53218: pacific: qa: Test failure: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable)ResolvedXiubo LiActions
Actions
Copy link
 #1

Updated by Xiubo Li over 2 years ago

  • Assignee set to Xiubo Li
Actions
Copy link
 #2

Updated by Xiubo Li over 2 years ago

  • Has duplicate Bug #52996: qa: test_perf_counters via test_openfiletable added
Actions
Copy link
 #3

Updated by Xiubo Li over 2 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #4

Updated by Xiubo Li over 2 years ago

The `self.wait_until_true(lambda: self._check_oft_counter('omap_total_removes', 1), timeout=30)` last check was at `2021-10-02T19:32:11.317`:

2021-10-02T19:32:11.317 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump
...
2021-10-02T19:32:11.493 DEBUG:tasks.cephfs.filesystem:_json_asok output
{
  ...
  "oft": {
    "omap_total_kv_pairs": 2,
    "omap_total_objs": 2,
    "omap_total_removes": 0,
    "omap_total_updates": 2
  },
  ...
}
2021-10-02T19:32:11.494 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules...
2021-10-02T19:32:11.495 DEBUG:teuthology.orchestra.run.smithi037:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'
2021-10-02T19:32:11.535 DEBUG:teuthology.orchestra.run.smithi101:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'
2021-10-02T19:32:11.579 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules.
2021-10-02T19:32:11.579 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0...
2021-10-02T19:32:11.580 INFO:teuthology.orchestra.run:Running command with timeout 900

But the `openfiles commit()` came just after that and 1 seconds later:


2021-10-02T19:31:12.406+0000 7f0d80837700 10 mds.0.openfiles commit log_seq 1

We should enlarge the wait time.

Actions
Copy link
 #5

Updated by Xiubo Li over 2 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 43767
Actions
Copy link
 #6

Updated by Patrick Donnelly over 2 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Target version set to v17.0.0
  • Source set to Q/A
  • Backport set to pacific
Actions
Copy link
 #7

Updated by Backport Bot over 2 years ago

  • Copied to Backport #53218: pacific: qa: Test failure: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable) added
Actions
Copy link
 #8

Updated by Loïc Dachary over 2 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions
Copy link

Also available in: Atom PDF