Actions
Bug #45190
openosd dump times out
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2020-04-20T13:04:39.254 INFO:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json
2020-04-20T13:06:39.334 DEBUG:teuthology.orchestra.run:got remote process result: 124
2020-04-20T13:06:39.335 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/contextutil.py", line 34, in nested
yield vars
File "/home/teuthworker/src/git.ceph.com_ceph_octopus/qa/tasks/ceph.py", line 1825, in task
cluster=config['cluster'],
File "/home/teuthworker/src/git.ceph.com_ceph_octopus/qa/tasks/ceph_manager.py", line 1328, in __init__
pools = self.list_pools()
File "/home/teuthworker/src/git.ceph.com_ceph_octopus/qa/tasks/ceph_manager.py", line 1721, in list_pools
osd_dump = self.get_osd_dump_json()
File "/home/teuthworker/src/git.ceph.com_ceph_octopus/qa/tasks/ceph_manager.py", line 2294, in get_osd_dump_json
out = self.raw_cluster_cmd('osd', 'dump', '--format=json')
File "/home/teuthworker/src/git.ceph.com_ceph_octopus/qa/tasks/ceph_manager.py", line 1362, in raw_cluster_cmd
stdout=BytesIO(),
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/remote.py", line 203, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/run.py", line 473, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/run.py", line 162, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/run.py", line 184, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi044 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json'
master - /a/bhubbard-2020-04-16_09:57:54-rados-wip-badone-testing-distro-basic-smithi/4957886/
octopus - /a/teuthology-2020-04-20_03:30:03-rados-octopus-distro-basic-smithi/4968388/
Updated by Brad Hubbard over 3 years ago
- Related to Bug #44631: ceph pg dump error code 124 added
Updated by Deepika Upadhyay over 3 years ago
/a/yuriw-2020-10-05_22:17:06-rados-wip-yuri7-testing-2020-10-05-1338-octopus-distro-basic-smithi/5500126/teuthology.log
020-10-07T18:14:19.530 INFO:tasks.ceph.osd.6.smithi192.stderr:2020-10-07T18:14:19.525+0000 7faf77146700 -1 osd.6 649 handle_osd_map injecting map crc failure
2020-10-07T18:14:21.848 INFO:tasks.ceph.osd.1.smithi101.stderr:2020-10-07T18:14:21.845+0000 7fbb34a48700 -1 osd.1 650 handle_osd_map injecting map crc failure
2020-10-07T18:14:41.162 INFO:tasks.ceph.osd.5.smithi192.stderr:2020-10-07T18:14:41.160+0000 7f22662fb700 -1 osd.5 651 handle_osd_map injecting map crc failure
2020-10-07T18:15:16.575 INFO:tasks.ceph.osd.5.smithi192.stderr:2020-10-07T18:15:16.572+0000 7f2268b00700 -1 osd.5 652 handle_osd_map injecting map crc failure
2020-10-07T18:15:16.890 INFO:tasks.ceph.osd.2.smithi101.stderr:2020-10-07T18:15:16.888+0000 7f93f04bd700 -1 osd.2 653 handle_osd_map injecting map crc failure
2020-10-07T18:15:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 124
2020-10-07T18:15:23.734 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested
yield vars
File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri7-testing-2020-10-05-1338-octopus/qa/tasks/ceph.py", line 1839, in task
osd_scrub_pgs(ctx, config)
File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri7-testing-2020-10-05-1338-octopus/qa/tasks/ceph.py", line 1193, in osd_scrub_pgs
stats = manager.get_pg_stats()
File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri7-testing-2020-10-05-1338-octopus/qa/tasks/ceph_manager.py", line 2132, in get_pg_stats
out = self.raw_cluster_cmd('pg', 'dump', '--format=json')
File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri7-testing-2020-10-05-1338-octopus/qa/tasks/ceph_manager.py", line 1363, in raw_cluster_cmd
stdout=BytesIO(),
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi101 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120
Updated by Neha Ojha over 3 years ago
2020-10-28T10:01:49.408 INFO:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-10-28T10:03:49.453 DEBUG:teuthology.orchestra.run:got remote process result: 124
/a/teuthology-2020-10-28_07:01:02-rados-master-distro-basic-smithi/5567403
Actions