Actions
Bug #47051
closedfs/upgrade/volume_client: Command failed with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json'
Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
Testing
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
fs
Component(FS):
qa-suite
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Hit the following error in fs/upgrade/volume_client test,
2020-08-19T16:11:42.075 INFO:tasks.ceph.mds.c.smithi062.stderr:2020-08-19 16:11:42.067 7f656496d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i c (PID: 12002) UID: 0 2020-08-19T16:11:42.075 INFO:tasks.ceph.mds.c.smithi062.stderr:2020-08-19 16:11:42.067 7f656496d700 -1 mds.c *** got signal Terminated *** 2020-08-19T16:11:42.075 INFO:tasks.ceph.mon.a.smithi062.stderr:2020-08-19T16:11:42.067+0000 7fb5260fd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 16677) UID: 0 2020-08-19T16:11:42.075 INFO:tasks.ceph.mon.a.smithi062.stderr:2020-08-19T16:11:42.067+0000 7fb5260fd700 -1 mon.a@0(leader) e2 *** Got Signal Terminated *** 2020-08-19T16:11:42.075 INFO:tasks.ceph.mon.c.smithi062.stderr:2020-08-19T16:11:42.071+0000 7f3987edf700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 16736) UID: 0 2020-08-19T16:11:42.076 INFO:tasks.ceph.mon.c.smithi062.stderr:2020-08-19T16:11:42.071+0000 7f3987edf700 -1 mon.c@2(peon) e2 *** Got Signal Terminated *** 2020-08-19T16:11:42.076 INFO:tasks.ceph.mon.b.smithi062.stderr:2020-08-19T16:11:42.071+0000 7f2d77518700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 16703) UID: 0 2020-08-19T16:11:42.076 INFO:tasks.ceph.mon.b.smithi062.stderr:2020-08-19T16:11:42.071+0000 7f2d77518700 -1 mon.b@1(peon) e2 *** Got Signal Terminated *** 2020-08-19T16:11:42.197 INFO:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-08-19T16:11:42.276 INFO:tasks.ceph.mgr.y.smithi062.stderr:daemon-helper: command crashed with signal 15
And later,
2020-08-19T16:13:42.262 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=3f7baa7951d94ae5976213c5402c7c16 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 93, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ajarr-wip-46163/qa/tasks/ceph.py", line 1551, in restart ctx.managers[cluster].wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_ajarr-wip-46163/qa/tasks/ceph_manager.py", line 2534, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_ajarr-wip-46163/qa/tasks/ceph_manager.py", line 2524, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_ajarr-wip-46163/qa/tasks/ceph_manager.py", line 2297, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_ajarr-wip-46163/qa/tasks/ceph_manager.py", line 2289, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_ajarr-wip-46163/qa/tasks/ceph_manager.py", line 1357, in raw_cluster_cmd stdout=StringIO(), File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 204, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json'
Possibly an issue with the setup and teardown of the tests?
Logs are here,
http://pulpito.front.sepia.ceph.com/rraja-2020-08-19_13:10:02-fs-ajarr-wip-46163-distro-basic-smithi/5359764/
http://pulpito.front.sepia.ceph.com/rraja-2020-08-19_13:10:02-fs-ajarr-wip-46163-distro-basic-smithi/5359765/
http://pulpito.front.sepia.ceph.com/rraja-2020-08-19_13:10:02-fs-ajarr-wip-46163-distro-basic-smithi/5359766/
Actions