Actions
Bug #63198
openrados/thrash: AssertionError: wait_for_recovery: failed before timeout expired
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
/a/ksirivad-2023-10-13_01:58:36-rados-wip-ksirivad-fix-63183-distro-default-smithi/7423809/teuthology.log
2023-10-13T05:29:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 124
2023-10-13T05:29:39.338 ERROR:teuthology.run_tasks:Manager failed: thrashosds
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
next(self.gen)
File "/home/teuthworker/src/github.com_ceph_ceph-c_5d1b5da21591c57cb0cbbbc8775b6ea0ced953a4/qa/tasks/thrashosds.py", line 215, in task
cluster_manager.wait_for_all_osds_up()
File "/home/teuthworker/src/github.com_ceph_ceph-c_5d1b5da21591c57cb0cbbbc8775b6ea0ced953a4/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up
while not self.are_all_osds_up():
File "/home/teuthworker/src/github.com_ceph_ceph-c_5d1b5da21591c57cb0cbbbc8775b6ea0ced953a4/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up
x = self.get_osd_dump()
File "/home/teuthworker/src/github.com_ceph_ceph-c_5d1b5da21591c57cb0cbbbc8775b6ea0ced953a4/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump
return self.get_osd_dump_json()['osds']
File "/home/teuthworker/src/github.com_ceph_ceph-c_5d1b5da21591c57cb0cbbbc8775b6ea0ced953a4/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json
out = self.raw_cluster_cmd('osd', 'dump', '--format=json')
File "/home/teuthworker/src/github.com_ceph_ceph-c_5d1b5da21591c57cb0cbbbc8775b6ea0ced953a4/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd
return self.run_cluster_cmd(**kwargs).stdout.getvalue()
File "/home/teuthworker/src/github.com_ceph_ceph-c_5d1b5da21591c57cb0cbbbc8775b6ea0ced953a4/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd
return self.controller.run(**kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi072 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'
2023-10-13T05:29:39.347 DEBUG:teuthology.run_tasks:Unwinding manager ceph
2023-10-13T05:29:39.356 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean
Actions