Actions
Bug #63587
openTest failure: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc)
Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
Testing
Target version:
% Done:
0%
Source:
Q/A
Tags:
backport_processed
Backport:
quincy,reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2023-11-16T04:49:25.624 INFO:teuthology.orchestra.run.smithi005.stderr:2023-11-16T04:49:25.619+0000 7f9ce68f1640 1 -- 172.21.15.5:0/2045590866 shutdown_connections 2023-11-16T04:49:25.624 INFO:teuthology.orchestra.run.smithi005.stderr:2023-11-16T04:49:25.619+0000 7f9ce68f1640 1 -- 172.21.15.5:0/2045590866 wait complete. 2023-11-16T04:49:25.646 INFO:tasks.cephfs_test_runner:To check whether the fsync will be stuck to wait for the mdlog to be ... FAIL 2023-11-16T04:49:25.646 INFO:tasks.cephfs_test_runner: 2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:FAIL: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc) 2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:To check whether the fsync will be stuck to wait for the mdlog to be 2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_cb2179fff8710af9ff3936b02b8bc8cd99900af3/qa/tasks/cephfs/test_misc.py", line 288, in test_filesystem_sync_stuck_for_around_5s 2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner: self._test_sync_stuck_for_around_5s(dir_path) 2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_cb2179fff8710af9ff3936b02b8bc8cd99900af3/qa/tasks/cephfs/test_misc.py", line 277, in _test_sync_stuck_for_around_5s 2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner: self.assertLess(duration, 4) 2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner:AssertionError: 6.773383140563965 not less than 4 2023-11-16T04:49:25.647 INFO:tasks.cephfs_test_runner: 2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:Ran 14 tests in 2185.087s 2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner: 2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:FAILED (failures=1) 2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner: 2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:FAIL: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc) 2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:To check whether the fsync will be stuck to wait for the mdlog to be 2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_cb2179fff8710af9ff3936b02b8bc8cd99900af3/qa/tasks/cephfs/test_misc.py", line 288, in test_filesystem_sync_stuck_for_around_5s 2023-11-16T04:49:25.648 INFO:tasks.cephfs_test_runner: self._test_sync_stuck_for_around_5s(dir_path) 2023-11-16T04:49:25.649 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_cb2179fff8710af9ff3936b02b8bc8cd99900af3/qa/tasks/cephfs/test_misc.py", line 277, in _test_sync_stuck_for_around_5s 2023-11-16T04:49:25.649 INFO:tasks.cephfs_test_runner: self.assertLess(duration, 4) 2023-11-16T04:49:25.649 INFO:tasks.cephfs_test_runner:AssertionError: 6.773383140563965 not less than 4 2023-11-16T04:49:25.649 INFO:tasks.cephfs_test_runner: 2023-11-16T04:49:25.649 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_cb2179fff8710af9ff3936b02b8bc8cd99900af3/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc) 2023-11-16T04:49:25.714 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=22b205cc57aa4163842515db3f0b6b5f Traceback (most recent call last):
Updated by Xiubo Li 6 months ago
2023-11-16T04:49:07.079 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir filesystem_sync_do_not_wait_mdlog_testdir/21_3) 2023-11-16T04:49:07.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T04:49:07.131 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir filesystem_sync_do_not_wait_mdlog_testdir/21_4) 2023-11-16T04:49:07.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T04:49:07.182 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec sync) 2023-11-16T04:49:07.291 INFO:tasks.cephfs.test_misc:sync mkdir i = 21, duration = 0.10925078392028809 2023-11-16T04:49:07.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T04:49:07.292 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf filesystem_sync_do_not_wait_mdlog_testdir/21_0) 2023-11-16T04:49:07.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T04:49:07.313 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf filesystem_sync_do_not_wait_mdlog_testdir/21_1) 2023-11-16T04:49:07.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T04:49:07.376 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf filesystem_sync_do_not_wait_mdlog_testdir/21_2) 2023-11-16T04:49:07.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T04:49:07.439 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf filesystem_sync_do_not_wait_mdlog_testdir/21_3) 2023-11-16T04:49:07.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T04:49:07.503 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf filesystem_sync_do_not_wait_mdlog_testdir/21_4) 2023-11-16T04:49:07.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T04:49:14.022 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.0 && exec sync) 2023-11-16T04:49:14.338 INFO:tasks.cephfs.test_misc:sync rmdir i = 21, duration = 6.773383140563965
We can see that the shell command exec sync was deferred to trigger. While the sync itself was very fast in 1s in the client.
Updated by Venky Shankar 3 months ago
- Category set to Testing
- Status changed from Fix Under Review to Pending Backport
- Target version set to v19.0.0
- Source set to Q/A
- Backport changed from pacific, quincy, reef to quincy,reef
Updated by Backport Bot 3 months ago
- Copied to Backport #64221: quincy: Test failure: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc) added
Updated by Backport Bot 3 months ago
- Copied to Backport #64222: reef: Test failure: test_filesystem_sync_stuck_for_around_5s (tasks.cephfs.test_misc.TestMisc) added
Actions