Bug #43718
Updated by Patrick Donnelly over 4 years ago
Edit: the real issue is that the exception causes you to think the test failed due to this. It does not. I've submitted a PR that squelches the traceback.
<pre>
2020-01-18T03:12:41.502 INFO:teuthology.orchestra.run.smithi008.stderr:Error EBUSY:
2020-01-18T03:12:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 16
2020-01-18T03:12:41.508 ERROR:tasks.mon_thrash:Saw exception while triggering scrub
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200117.183318/qa/tasks/mon_thrash.py", line 332, in _do_thrash
self.manager.raw_cluster_cmd('mon', 'scrub')
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200117.183318/qa/tasks/ceph_manager.py", line 1306, in raw_cluster_cmd
stdout=StringIO(),
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 433, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi008 with status 16: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub'
</pre>
From: /ceph/teuthology-archive/pdonnell-2020-01-17_22:34:19-fs-wip-pdonnell-testing-20200117.183318-distro-basic-smithi/4679523/teuthology.log
mon log:
<pre>
2020-01-18T03:12:41.491+0000 7fbad4ee3700 0 log_channel(audit) log [INF] : from='client.874245 172.21.15.8:0/2694360193' entity='client.admin' cmd=[{"prefix": "mon scrub"}]: dispatch
2020-01-18T03:12:41.491+0000 7fbad4ee3700 1 -- [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- log(1 entries from seq 70 at 2020-01-18T03:12:41.492339+0000) v1 -- 0x557ebcef2480 con 0x557eb9932400
2020-01-18T03:12:41.491+0000 7fbad4ee3700 10 mon.a@0(leader) e1 scrub_start
2020-01-18T03:12:41.491+0000 7fbad4ee3700 0 log_channel(cluster) log [INF] : scrub already in progress
</pre>
From: /ceph/teuthology-archive/pdonnell-2020-01-17_22:34:19-fs-wip-pdonnell-testing-20200117.183318-distro-basic-smithi/4679523/remote/smithi008/log/ceph-mon.a.log.gz