Project

General

Profile

Actions
Copy link

Bug #63587

open

Test failure: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc)

Added by Xiubo Li 5 months ago. Updated 3 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Xiubo Li
Category:
Testing
Target version:
Ceph - v19.0.0
% Done:

0%

Source:
Q/A
Tags:
backport_processed
Backport:
quincy,reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa, qa-failure
Pull request ID:
54683
Crash signature (v1):
Crash signature (v2):

Description

https://pulpito.ceph.com/yuriw-2023-11-15_23:07:34-fs-wip-yuri-testing-2023-11-14-0743-reef-distro-default-smithi/7460053/

2023-11-16T04:49:25.624 INFO:teuthology.orchestra.run.smithi005.stderr:2023-11-16T04:49:25.619+0000 7f9ce68f1640  1 -- 172.21.15.5:0/2045590866 shutdown_connections
2023-11-16T04:49:25.624 INFO:teuthology.orchestra.run.smithi005.stderr:2023-11-16T04:49:25.619+0000 7f9ce68f1640  1 -- 172.21.15.5:0/2045590866 wait complete.
2023-11-16T04:49:25.646 INFO:tasks.cephfs_test_runner:To check whether the fsync will be stuck to wait for the mdlog to be ... FAIL
2023-11-16T04:49:25.646 INFO:tasks.cephfs_test_runner:
2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:======================================================================
2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:FAIL: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc)
2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:To check whether the fsync will be stuck to wait for the mdlog to be
2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_cb2179fff8710af9ff3936b02b8bc8cd99900af3/qa/tasks/cephfs/test_misc.py", line 288, in test_filesystem_sync_stuck_for_around_5s
2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:    self._test_sync_stuck_for_around_5s(dir_path)
2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_cb2179fff8710af9ff3936b02b8bc8cd99900af3/qa/tasks/cephfs/test_misc.py", line 277, in _test_sync_stuck_for_around_5s
2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:    self.assertLess(duration, 4)
2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:AssertionError: 6.773383140563965 not less than 4
2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:
2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:Ran 14 tests in 2185.087s
2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:
2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:FAILED (failures=1)
2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:
2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:======================================================================
2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:FAIL: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc)
2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:To check whether the fsync will be stuck to wait for the mdlog to be
2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_cb2179fff8710af9ff3936b02b8bc8cd99900af3/qa/tasks/cephfs/test_misc.py", line 288, in test_filesystem_sync_stuck_for_around_5s
2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:    self._test_sync_stuck_for_around_5s(dir_path)
2023-11-16T04:49:25.649 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_cb2179fff8710af9ff3936b02b8bc8cd99900af3/qa/tasks/cephfs/test_misc.py", line 277, in _test_sync_stuck_for_around_5s
2023-11-16T04:49:25.649 INFO:tasks.cephfs_test_runner:    self.assertLess(duration, 4)
2023-11-16T04:49:25.649 INFO:tasks.cephfs_test_runner:AssertionError: 6.773383140563965 not less than 4
2023-11-16T04:49:25.649 INFO:tasks.cephfs_test_runner:
2023-11-16T04:49:25.649 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_cb2179fff8710af9ff3936b02b8bc8cd99900af3/qa/tasks/cephfs_test_runner.py", line 211, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc)
2023-11-16T04:49:25.714 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=22b205cc57aa4163842515db3f0b6b5f
Traceback (most recent call last):

Related issues 2 (2 open — 0 closed)

Copied to CephFS - Backport #64221: quincy: Test failure: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc)In ProgressXiubo LiActions
Copied to CephFS - Backport #64222: reef: Test failure: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc)In ProgressXiubo LiActions
Actions
Copy link
 #1

Updated by Xiubo Li 5 months ago 

2023-11-16T04:49:07.079 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir filesystem_sync_do_not_wait_mdlog_testdir/21_3)
2023-11-16T04:49:07.130 INFO:teuthology.orchestra.run:Running command with timeout 300
2023-11-16T04:49:07.131 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir filesystem_sync_do_not_wait_mdlog_testdir/21_4)
2023-11-16T04:49:07.182 INFO:teuthology.orchestra.run:Running command with timeout 300
2023-11-16T04:49:07.182 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec sync)
2023-11-16T04:49:07.291 INFO:tasks.cephfs.test_misc:sync mkdir i = 21, duration = 0.10925078392028809
2023-11-16T04:49:07.292 INFO:teuthology.orchestra.run:Running command with timeout 300
2023-11-16T04:49:07.292 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf filesystem_sync_do_not_wait_mdlog_testdir/21_0)
2023-11-16T04:49:07.313 INFO:teuthology.orchestra.run:Running command with timeout 300
2023-11-16T04:49:07.313 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf filesystem_sync_do_not_wait_mdlog_testdir/21_1)
2023-11-16T04:49:07.376 INFO:teuthology.orchestra.run:Running command with timeout 300
2023-11-16T04:49:07.376 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf filesystem_sync_do_not_wait_mdlog_testdir/21_2)
2023-11-16T04:49:07.439 INFO:teuthology.orchestra.run:Running command with timeout 300
2023-11-16T04:49:07.439 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf filesystem_sync_do_not_wait_mdlog_testdir/21_3)
2023-11-16T04:49:07.503 INFO:teuthology.orchestra.run:Running command with timeout 300
2023-11-16T04:49:07.503 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf filesystem_sync_do_not_wait_mdlog_testdir/21_4)
2023-11-16T04:49:07.564 INFO:teuthology.orchestra.run:Running command with timeout 300
2023-11-16T04:49:14.022 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec sync)
2023-11-16T04:49:14.338 INFO:tasks.cephfs.test_misc:sync rmdir i = 21, duration = 6.773383140563965

We can see that the shell command exec sync was deferred to trigger. While the sync itself was very fast in 1s in the client.

Actions
Copy link
 #2

Updated by Xiubo Li 5 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 54683
Actions
Copy link
 #3

Updated by Venky Shankar 3 months ago

  • Category set to Testing
  • Status changed from Fix Under Review to Pending Backport
  • Target version set to v19.0.0
  • Source set to Q/A
  • Backport changed from pacific, quincy, reef to quincy,reef
Actions
Copy link
 #4

Updated by Venky Shankar 3 months ago

  • Labels (FS) qa, qa-failure added
Actions
Copy link
 #5

Updated by Backport Bot 3 months ago

  • Copied to Backport #64221: quincy: Test failure: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc) added
Actions
Copy link
 #6

Updated by Backport Bot 3 months ago

  • Copied to Backport #64222: reef: Test failure: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc) added
Actions
Copy link
 #7

Updated by Backport Bot 3 months ago

  • Tags set to backport_processed
Actions
Copy link

Also available in: Atom PDF