Project

General

Profile

Bug #36107

mgr/dashboard: Fix "randomly" failing QA test "test_scrub"

Added by Patrick Seidensal 11 months ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Category:
dashboard/general
Target version:
-
Start date:
09/21/2018
Due date:
% Done:

100%

Source:
Tags:
Backport:
Regression:
Yes
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

Test failure: test_scrub (tasks.mgr.dashboard.test_osd.OsdTest)

[...]
2018-09-21T07:25:07.646 INFO:tasks.mgr.dashboard.helper:request GET to https://smithi165.front.sepia.ceph.com:7789/api/osd/13/safe_to_destroy
2018-09-21T07:25:07.653 INFO:tasks.ceph.mgr.x.smithi165.stdout:::ffff:172.21.0.51 - - [21/Sep/2018:07:25:07] "GET /api/osd/13/safe_to_destroy HTTP/1.1" 200 25 "" "python-requests/2.12.5" 
2018-09-21T07:25:07.657 INFO:tasks.mgr.dashboard.helper:request GET to https://smithi165.front.sepia.ceph.com:7789/api/osd/0/safe_to_destroy
2018-09-21T07:25:07.663 INFO:tasks.ceph.mgr.x.smithi165.stderr:2018-09-21 07:25:07.667 7f4042566700 -1 mgr.server reply reply (16) Device or resource busy OSD(s) 0 have 88 pgs currently mapped to them
2018-09-21T07:25:07.664 INFO:tasks.ceph.mgr.x.smithi165.stdout:::ffff:172.21.0.51 - - [21/Sep/2018:07:25:07] "GET /api/osd/0/safe_to_destroy HTTP/1.1" 200 86 "" "python-requests/2.12.5" 
2018-09-21T07:25:07.665 INFO:tasks.mgr.dashboard.helper:request POST to https://smithi165.front.sepia.ceph.com:7789/api/osd/0/mark_in
2018-09-21T07:25:07.673 INFO:tasks.ceph.mgr.x.smithi165.stdout:::ffff:172.21.0.51 - - [21/Sep/2018:07:25:07] "POST /api/osd/0/mark_in HTTP/1.1" 200 4 "" "python-requests/2.12.5" 
2018-09-21T07:25:07.674 INFO:tasks.cephfs_test_runner:test_safe_to_destroy (tasks.mgr.dashboard.test_osd.OsdTest) ... ok
2018-09-21T07:25:07.675 INFO:tasks.cephfs_test_runner:Starting test: test_scrub (tasks.mgr.dashboard.test_osd.OsdTest)
2018-09-21T07:25:07.675 INFO:tasks.mgr.dashboard.helper:request POST to https://smithi165.front.sepia.ceph.com:7789/api/osd/0/scrub?deep=False
2018-09-21T07:25:07.681 INFO:tasks.ceph.mgr.x.smithi165.stderr:2018-09-21 07:25:07.683 7f4042566700 -1 mgr.server reply reply (11) Resource temporarily unavailable osd.0 is not up
2018-09-21T07:25:07.765 INFO:tasks.cephfs_test_runner:test_scrub (tasks.mgr.dashboard.test_osd.OsdTest) ... FAIL
2018-09-21T07:25:07.765 INFO:tasks.mgr.dashboard.helper:request POST to https://smithi165.front.sepia.ceph.com:7789/api/osd/0/mark_in
2018-09-21T07:25:07.767 INFO:tasks.ceph.mgr.x.smithi165.stdout:::ffff:172.21.0.51 - - [21/Sep/2018:07:25:07] "POST /api/osd/0/scrub?deep=False HTTP/1.1" 400 88 "" "python-requests/2.12.5" 
2018-09-21T07:25:07.774 INFO:tasks.ceph.mgr.x.smithi165.stdout:::ffff:172.21.0.51 - - [21/Sep/2018:07:25:07] "POST /api/osd/0/mark_in HTTP/1.1" 200 4 "" "python-requests/2.12.5" 
2018-09-21T07:25:07.775 INFO:tasks.cephfs_test_runner:
2018-09-21T07:25:07.775 INFO:tasks.cephfs_test_runner:======================================================================
2018-09-21T07:25:07.776 INFO:tasks.cephfs_test_runner:FAIL: test_scrub (tasks.mgr.dashboard.test_osd.OsdTest)
2018-09-21T07:25:07.776 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2018-09-21T07:25:07.776 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2018-09-21T07:25:07.776 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-23568-2/qa/tasks/mgr/dashboard/test_osd.py", line 52, in test_scrub
2018-09-21T07:25:07.776 INFO:tasks.cephfs_test_runner:    self.assertStatus(200)
2018-09-21T07:25:07.776 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-23568-2/qa/tasks/mgr/dashboard/helper.py", line 325, in assertStatus
2018-09-21T07:25:07.776 INFO:tasks.cephfs_test_runner:    self.assertEqual(self._resp.status_code, status)
2018-09-21T07:25:07.776 INFO:tasks.cephfs_test_runner:AssertionError: 400 != 200
2018-09-21T07:25:07.776 INFO:tasks.cephfs_test_runner:
2018-09-21T07:25:07.776 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2018-09-21T07:25:07.776 INFO:tasks.cephfs_test_runner:Ran 37 tests in 366.436s
2018-09-21T07:25:07.777 INFO:tasks.cephfs_test_runner:
2018-09-21T07:25:07.777 INFO:tasks.cephfs_test_runner:FAILED (failures=1)
2018-09-21T07:25:07.777 INFO:tasks.cephfs_test_runner:
2018-09-21T07:25:07.777 INFO:tasks.cephfs_test_runner:======================================================================
2018-09-21T07:25:07.777 INFO:tasks.cephfs_test_runner:FAIL: test_scrub (tasks.mgr.dashboard.test_osd.OsdTest)
2018-09-21T07:25:07.777 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2018-09-21T07:25:07.777 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2018-09-21T07:25:07.777 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-23568-2/qa/tasks/mgr/dashboard/test_osd.py", line 52, in test_scrub
2018-09-21T07:25:07.777 INFO:tasks.cephfs_test_runner:    self.assertStatus(200)
2018-09-21T07:25:07.777 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-23568-2/qa/tasks/mgr/dashboard/helper.py", line 325, in assertStatus
2018-09-21T07:25:07.777 INFO:tasks.cephfs_test_runner:    self.assertEqual(self._resp.status_code, status)
2018-09-21T07:25:07.778 INFO:tasks.cephfs_test_runner:AssertionError: 400 != 200
2018-09-21T07:25:07.778 INFO:tasks.cephfs_test_runner:
2018-09-21T07:25:07.778 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lpaduano-testing-23568-2/qa/tasks/cephfs_test_runner.py", line 207, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_scrub (tasks.mgr.dashboard.test_osd.OsdTest)
2018-09-21T07:25:07.801 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=f385076654f841d7836070e5ec7ffb0a
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
    manager.__enter__()

[...]

History

#1 Updated by Patrick Seidensal 11 months ago

  • Subject changed from Dashboard: Fix randomly failing QA test "test_scrub" to Dashboard: Fix "randomly" failing QA test "test_scrub"

#3 Updated by Lenz Grimmer 11 months ago

  • Subject changed from Dashboard: Fix "randomly" failing QA test "test_scrub" to mgr/dashboard: Fix "randomly" failing QA test "test_scrub"

#4 Updated by Patrick Seidensal 11 months ago

  • Status changed from Need Review to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF