Project

General

Profile

Bug #43718

qa/tasks/mon_thrash: hide traceback from mon scrub failures

Added by Patrick Donnelly 6 months ago. Updated 6 months ago.

Status:
Pending Backport
Priority:
High
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
nautilus,mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

Edit: the real issue is that the exception causes you to think the test failed due to this. It does not. I've submitted a PR that squelches the traceback.

2020-01-18T03:12:41.502 INFO:teuthology.orchestra.run.smithi008.stderr:Error EBUSY:
2020-01-18T03:12:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 16
2020-01-18T03:12:41.508 ERROR:tasks.mon_thrash:Saw exception while triggering scrub
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200117.183318/qa/tasks/mon_thrash.py", line 332, in _do_thrash
    self.manager.raw_cluster_cmd('mon', 'scrub')
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200117.183318/qa/tasks/ceph_manager.py", line 1306, in raw_cluster_cmd
    stdout=StringIO(),
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 433, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status
    node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi008 with status 16: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub'

From: /ceph/teuthology-archive/pdonnell-2020-01-17_22:34:19-fs-wip-pdonnell-testing-20200117.183318-distro-basic-smithi/4679523/teuthology.log

mon log:

2020-01-18T03:12:41.491+0000 7fbad4ee3700  0 log_channel(audit) log [INF] : from='client.874245 172.21.15.8:0/2694360193' entity='client.admin' cmd=[{"prefix": "mon scrub"}]: dispatch
2020-01-18T03:12:41.491+0000 7fbad4ee3700  1 -- [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- log(1 entries from seq 70 at 2020-01-18T03:12:41.492339+0000) v1 -- 0x557ebcef2480 con 0x557eb9932400
2020-01-18T03:12:41.491+0000 7fbad4ee3700 10 mon.a@0(leader) e1 scrub_start
2020-01-18T03:12:41.491+0000 7fbad4ee3700  0 log_channel(cluster) log [INF] : scrub already in progress

From: /ceph/teuthology-archive/pdonnell-2020-01-17_22:34:19-fs-wip-pdonnell-testing-20200117.183318-distro-basic-smithi/4679523/remote/smithi008/log/ceph-mon.a.log.gz


Related issues

Copied to Ceph - Backport #43773: nautilus: qa/tasks/mon_thrash: hide traceback from mon scrub failures Resolved
Copied to Ceph - Backport #43774: mimic: qa/tasks/mon_thrash: hide traceback from mon scrub failures New

History

#1 Updated by Patrick Donnelly 6 months ago

  • Subject changed from qa/tasks/mon_thrash: tolerate EBUSY for mon scrub to qa/tasks/mon_thrash: hide traceback from mon scrub failures
  • Description updated (diff)
  • Status changed from New to Fix Under Review
  • Pull request ID set to 32739

#2 Updated by Patrick Donnelly 6 months ago

  • Status changed from Fix Under Review to Pending Backport

#3 Updated by Nathan Cutler 6 months ago

  • Copied to Backport #43773: nautilus: qa/tasks/mon_thrash: hide traceback from mon scrub failures added

#4 Updated by Nathan Cutler 6 months ago

  • Copied to Backport #43774: mimic: qa/tasks/mon_thrash: hide traceback from mon scrub failures added

Also available in: Atom PDF