Bug #55805
Updated by Rishabh Dave almost 2 years ago
Bug discovered on QA run for PR - https://github.com/ceph/ceph/pull/45556
Teuthology job - https://pulpito.ceph.com/vshankar-2022-04-26_06:23:29-fs:workload-wip-45556-20220418-102656-testing-default-smithi/6806484/
workload: kernel_untar_build.yaml
Traceback #1 -
<pre>
2022-04-26T07:46:30.191 INFO:tasks.cephfs.filesystem:scrub status for tag:3f516427-181e-4cf8-a57d-669202f3a4f5 - {'path': '/', 'tag': '3f516427-181e-4cf8-a57d-669202f3a4f5', 'options': 'recursive,force'}
2022-04-26T07:46:30.192 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception:
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/fwd_scrub.py", line 38, in _run
self.do_scrub()
File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/fwd_scrub.py", line 55, in do_scrub
self._scrub()
File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/fwd_scrub.py", line 77, in _scrub
timeout=self.scrub_timeout)
File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/cephfs/filesystem.py", line 1617, in wait_until_scrub_complete
while proceed():
File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/contextutil.py", line 133, in __call__
raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (30) after waiting for 900 seconds
</pre>
Failing command traceback #1 - @2022-04-26T07:46:29.709 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status@
Traceback #2 -
<pre>
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/run_tasks.py", line 188, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/fwd_scrub.py", line 151, in task
stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers)
File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/fwd_scrub.py", line 86, in stop_all_fwd_scrubbers
raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}")
RuntimeError: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds
</pre>
Failing command traceback #2 - @2022-04-26T08:51:10.308 DEBUG:teuthology.orchestra.run.smithi138:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:1ccbc711b8876e630c0358e1d8d923daa34dca1e shell --fsid f2662818-c530-11ec-8c39-001a4aab830c -- ceph daemon mds.l perf dump@
Traceback #3 -
<pre>
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/contextutil.py", line 33, in nested
yield vars
File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/cephadm.py", line 1595, in task
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/run_tasks.py", line 188, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/fwd_scrub.py", line 151, in task
stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers)
File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/fwd_scrub.py", line 86, in stop_all_fwd_scrubbers
raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}")
RuntimeError: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds
</pre>
Traceback #4 -
<pre>
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/contextutil.py", line 33, in nested
yield vars
File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/task/install/__init__.py", line 619, in task
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/run_tasks.py", line 188, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/fwd_scrub.py", line 151, in task
stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers)
File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/fwd_scrub.py", line 86, in stop_all_fwd_scrubbers
raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}")
RuntimeError: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds
</pre>