Bug #59350
closedqa: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks) ... ERROR
100%
Description
2023-04-05T20:00:37.684 INFO:tasks.cephfs_test_runner:======================================================================
2023-04-05T20:00:37.685 INFO:tasks.cephfs_test_runner:ERROR: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks)
2023-04-05T20:00:37.685 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2023-04-05T20:00:37.685 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2023-04-05T20:00:37.685 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_adb401e638b5b52544c2d2c3633d8d6b2f1425f5/qa/tasks/cephfs/test_scrub_checks.py", line 376, in test_flag_scrub_mdsdir
2023-04-05T20:00:37.685 INFO:tasks.cephfs_test_runner: self.scrub_with_stray_evaluation(self.fs, self.mount_a, "/",
2023-04-05T20:00:37.686 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_adb401e638b5b52544c2d2c3633d8d6b2f1425f5/qa/tasks/cephfs/test_scrub_checks.py", line 323, in scrub_with_stray_evaluation
2023-04-05T20:00:37.686 INFO:tasks.cephfs_test_runner: fs.wait_until_scrub_complete(tag=out_json["scrub_tag"]), True)
2023-04-05T20:00:37.686 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_adb401e638b5b52544c2d2c3633d8d6b2f1425f5/qa/tasks/cephfs/filesystem.py", line 1673, in wait_until_scrub_complete
2023-04-05T20:00:37.686 INFO:tasks.cephfs_test_runner: status = out_json['scrubs'][tag]
2023-04-05T20:00:37.686 INFO:tasks.cephfs_test_runner:KeyError: 'd23e59d8-b522-48a9-b16d-0c5b03be8ac0'
2023-04-05T20:00:37.687 INFO:tasks.cephfs_test_runner:
2023-04-05T20:00:37.687 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/run_tasks.py", line 109, in run_tasks
manager.__enter__()
File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/home/teuthworker/src/github.com_ceph_ceph-c_adb401e638b5b52544c2d2c3633d8d6b2f1425f5/qa/tasks/cephfs_test_runner.py", line 211, in task
raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks)
2023-04-05T20:00:37.759 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a553dd00be254fde9aa4c77990709c9e
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/run_tasks.py", line 109, in run_tasks
manager.__enter__()
File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/home/teuthworker/src/github.com_ceph_ceph-c_adb401e638b5b52544c2d2c3633d8d6b2f1425f5/qa/tasks/cephfs_test_runner.py", line 211, in task
raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks)
2023-04-05T20:00:37.761 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner
2023-04-05T20:00:37.778 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse
2023-04-05T20:00:37.792 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients...
2023-04-05T20:00:37.793 DEBUG:teuthology.orchestra.run.smithi143:> set -ex
2023-04-05T20:00:37.793 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/proc/self/mounts of=/dev/stdout
2023-04-05T20:00:37.823 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is:
sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
Updated by Venky Shankar about 1 year ago
- Category set to Testing
- Assignee set to Dhairya Parmar
- Target version set to v19.0.0
Updated by Dhairya Parmar about 1 year ago
This shouldn't have failed as it has been tested by me(multiple times both on vstart as well teuthology) as well venky. I tried running scrub tests on main branch, surprisingly my code is yet not there in main o_0? http://pulpito.front.sepia.ceph.com/dparmar-2023-04-13_18:02:30-fs:functional-main-distro-default-smithi/. FYI this is not so old main build, its 34fc77ad585ad35dd6c0bbeea53b6fcd6b741571 which is just a day old
Updated by Dhairya Parmar about 1 year ago
I found an identical job from my run last month that passed with ease
Unfortunately, logs are gone but this run was for making sure test_stray_evaluation_with_scrub and test_flag_scrub_mdsdir pass.
Updated by Dhairya Parmar about 1 year ago
Latest run of tasks/scrub http://pulpito.front.sepia.ceph.com/dparmar-2023-04-14_12:28:32-fs:functional-wip-dparmar-MDS-53724-distro-default-smithi/
Failures are unrelated(rhel jobs failing at kernel package installation), other jobs are all green with ease
Updated by Dhairya Parmar about 1 year ago
cmd scrub status dumped following JSON:
{
"status": "scrub active (757 inodes in the stack)",
"scrubs": {}
}
while it should've something like this:
{
"status": "scrub active (757 inodes in the stack)",
"scrubs": {
"d23e59d8-b522-48a9-b16d-0c5b03be8ac0": {
"path": "/",
"tag": "d23e59d8-b522-48a9-b16d-0c5b03be8ac0",
"options": "scrub_mdsdir"
}
}
}
therefore it can't find key d23e59d8-b522-48a9-b16d-0c5b03be8ac0 and throws KeyError, I checked logs but everything seems to be normal, also this patch has been tested multiple times on teuthology as well locally. This shouldn't have failed. Recent run logs can be found in the above comment. I also fetched latest main branch in order to see if there was any change made to the code that broke but it still ran fine.
Updated by Dhairya Parmar almost 1 year ago
ran yuri's branch on fs suite with scrub yaml: http://pulpito.front.sepia.ceph.com/dparmar-2023-04-17_19:11:31-fs:functional-wip-yuri5-testing-2023-04-04-0814-distro-default-smithi/
3 dead, 1 failed but here my testcases passed, while 2 passed
Updated by Venky Shankar almost 1 year ago
Dhairya Parmar wrote:
cmd scrub status dumped following JSON:
[...]
while it should've something like this:
[...]therefore it can't find key d23e59d8-b522-48a9-b16d-0c5b03be8ac0 and throws KeyError, I checked logs but everything seems to be normal, also this patch has been tested multiple times on teuthology as well locally. This shouldn't have failed. Recent run logs can be found in the above comment. I also fetched latest main branch in order to see if there was any change made to the code that broke but it still ran fine.
This seems like a race where the scrub tags are not listed.
Updated by Dhairya Parmar almost 1 year ago
Venky Shankar wrote:
Dhairya Parmar wrote:
cmd scrub status dumped following JSON:
[...]
while it should've something like this:
[...]therefore it can't find key d23e59d8-b522-48a9-b16d-0c5b03be8ac0 and throws KeyError, I checked logs but everything seems to be normal, also this patch has been tested multiple times on teuthology as well locally. This shouldn't have failed. Recent run logs can be found in the above comment. I also fetched latest main branch in order to see if there was any change made to the code that broke but it still ran fine.
This seems like a race where the scrub tags are not listed.
yeah, this wasn't seen before
Updated by Venky Shankar 11 months ago
Updated by Dhairya Parmar 11 months ago
- Status changed from Triaged to Fix Under Review
Updated by Venky Shankar 9 months ago
- Status changed from Fix Under Review to Pending Backport
- Backport set to reef,quincy,pacific
- Component(FS) MDS added
- Labels (FS) scrub added
Updated by Backport Bot 9 months ago
- Copied to Backport #62068: pacific: qa: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks) ... ERROR added
Updated by Backport Bot 9 months ago
- Copied to Backport #62069: reef: qa: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks) ... ERROR added
Updated by Backport Bot 9 months ago
- Copied to Backport #62070: quincy: qa: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks) ... ERROR added
Updated by Konstantin Shalygin 8 months ago
- Status changed from Pending Backport to Resolved
- % Done changed from 0 to 100