Actions
Bug #48680
openmds: scrubbing stuck "scrub active (0 inodes in the stack)"
% Done:
0%
Source:
Q/A
Tags:
Backport:
quincy, pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2020-12-18T02:16:25.467 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2020-12-18T02:16:25.467 INFO:teuthology.orchestra.run.smithi193.stdout: "status": "scrub active (0 inodes in the stack)", 2020-12-18T02:16:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: "scrubs": { 2020-12-18T02:16:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: "e366f5ff-325e-460c-b4c7-e6095d429c92": { 2020-12-18T02:16:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: "path": "/", 2020-12-18T02:16:25.468 INFO:teuthology.orchestra.run.smithi193.stdout: "tag": "e366f5ff-325e-460c-b4c7-e6095d429c92", 2020-12-18T02:16:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: "options": "recursive,force" 2020-12-18T02:16:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: } 2020-12-18T02:16:25.469 INFO:teuthology.orchestra.run.smithi193.stdout: } 2020-12-18T02:16:25.469 INFO:teuthology.orchestra.run.smithi193.stdout:} 2020-12-18T02:16:25.470 INFO:tasks.fwd_scrub.fs.[cephfs]:scrub status for tag:e366f5ff-325e-460c-b4c7-e6095d429c92 - {'path': '/', 'tag': 'e366f5ff-325e-460c-b4c7-e6095d429c92', 'options': 'recursive,force'} 2020-12-18T02:16:25.470 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201217.205941/qa/tasks/fwd_scrub.py", line 32, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201217.205941/qa/tasks/fwd_scrub.py", line 50, in do_scrub self.wait_until_scrub_complete(tag) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201217.205941/qa/tasks/fwd_scrub.py", line 55, in wait_until_scrub_complete while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (10) after waiting for 300 seconds 2020-12-18T02:16:25.470 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201217.205941/qa/tasks/fwd_scrub.py", line 100, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201217.205941/qa/tasks/fwd_scrub.py", line 144, in do_scrub raise RuntimeError('error during scrub thrashing') RuntimeError: error during scrub thrashing 2020-12-18T02:16:28.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~280s 2020-12-18T02:16:28.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~266s 2020-12-18T02:16:28.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~259s 2020-12-18T02:16:28.198 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.fs.[cephfs] failed 2020-12-18T02:16:28.198 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons
From: /ceph/teuthology-archive/pdonnell-2020-12-17_23:13:08-fs-wip-pdonnell-testing-20201217.205941-distro-basic-smithi/5715913/teuthology.log
Actions