Project

General

Profile

Bug #40618

Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress)

Added by Kefu Chai 9 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

2019-07-02T06:37:28.073 INFO:tasks.cephfs_test_runner:FAIL: test_osd_came_back (tasks.mgr.test_progress.TestProgress)
2019-07-02T06:37:28.073 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-07-02T06:37:28.073 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-07-02T06:37:28.073 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-06-28-1816/qa/tasks/mgr/test_progress.py", line 176, in test_osd_came_back
2019-07-02T06:37:28.073 INFO:tasks.cephfs_test_runner:    self.assertTrue(self._is_quiet())
2019-07-02T06:37:28.073 INFO:tasks.cephfs_test_runner:AssertionError: False is not true
2019-07-02T06:37:28.073 INFO:tasks.cephfs_test_runner:
2019-07-02T06:37:28.074 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-07-02T06:37:28.074 INFO:tasks.cephfs_test_runner:Ran 1 test in 71.945s
2019-07-02T06:37:28.074 INFO:tasks.cephfs_test_runner:
2019-07-02T06:37:28.074 INFO:tasks.cephfs_test_runner:FAILED (failures=1)
2019-07-02T06:37:25.213 INFO:teuthology.orchestra.run.mira030.stderr:2019-07-02T06:37:25.206+0000 7fa7c6397700  1 -- 172.21.4.116:0/1613632094 --> [v2:172.21.4.116:6800/14006,v1:172.21.4.116:6801/14006] -- command(tid 0: {"prefix": "progress json", "target": ["mgr", ""]})
 v1 -- 0x7fa7c016e030 con 0x7fa7a4033b00
2019-07-02T06:37:25.214 INFO:teuthology.orchestra.run.mira030.stderr:2019-07-02T06:37:25.210+0000 7fa7bcff9700  1 -- 172.21.4.116:0/1613632094 <== mgr.4620 v2:172.21.4.116:6800/14006 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+434 (crc 0 0 0) 0x7fa7b0000fb0 con 0x7fa7a403
3b00
2019-07-02T06:37:25.214 INFO:teuthology.orchestra.run.mira030.stderr:2019-07-02T06:37:25.210+0000 7fa7bcff9700 20 mgrc handle_command_reply command_reply(tid 0: 0 ) v1
2019-07-02T06:37:25.216 INFO:teuthology.orchestra.run.mira030.stdout:{
2019-07-02T06:37:25.216 INFO:teuthology.orchestra.run.mira030.stdout:  "completed": [
2019-07-02T06:37:25.216 INFO:teuthology.orchestra.run.mira030.stdout:    {
2019-07-02T06:37:25.216 INFO:teuthology.orchestra.run.mira030.stdout:      "message": "Rebalancing after osd.0 marked out",
2019-07-02T06:37:25.216 INFO:teuthology.orchestra.run.mira030.stdout:      "id": "6aec1958-769a-44b5-bd49-ecd619eb65af",
2019-07-02T06:37:25.216 INFO:teuthology.orchestra.run.mira030.stdout:      "refs": [
2019-07-02T06:37:25.216 INFO:teuthology.orchestra.run.mira030.stdout:        [
2019-07-02T06:37:25.217 INFO:teuthology.orchestra.run.mira030.stdout:          "osd",
2019-07-02T06:37:25.217 INFO:teuthology.orchestra.run.mira030.stdout:          0
2019-07-02T06:37:25.217 INFO:teuthology.orchestra.run.mira030.stdout:        ]
2019-07-02T06:37:25.217 INFO:teuthology.orchestra.run.mira030.stdout:      ]
2019-07-02T06:37:25.217 INFO:teuthology.orchestra.run.mira030.stdout:    }
2019-07-02T06:37:25.217 INFO:teuthology.orchestra.run.mira030.stdout:  ],
2019-07-02T06:37:25.217 INFO:teuthology.orchestra.run.mira030.stdout:  "events": [
2019-07-02T06:37:25.218 INFO:teuthology.orchestra.run.mira030.stdout:    {
2019-07-02T06:37:25.218 INFO:teuthology.orchestra.run.mira030.stdout:      "message": "Rebalancing after osd.0 marked in",
2019-07-02T06:37:25.218 INFO:teuthology.orchestra.run.mira030.stdout:      "id": "2416d7fe-52e8-4ab8-98ac-0b307388e2fd",
2019-07-02T06:37:25.218 INFO:teuthology.orchestra.run.mira030.stdout:      "refs": [
2019-07-02T06:37:25.218 INFO:teuthology.orchestra.run.mira030.stdout:        [
2019-07-02T06:37:25.218 INFO:teuthology.orchestra.run.mira030.stdout:          "osd",
2019-07-02T06:37:25.218 INFO:teuthology.orchestra.run.mira030.stdout:          0
2019-07-02T06:37:25.219 INFO:teuthology.orchestra.run.mira030.stdout:        ]
2019-07-02T06:37:25.219 INFO:teuthology.orchestra.run.mira030.stdout:      ]
2019-07-02T06:37:25.219 INFO:teuthology.orchestra.run.mira030.stdout:    }
2019-07-02T06:37:25.219 INFO:teuthology.orchestra.run.mira030.stdout:  ]
2019-07-02T06:37:25.219 INFO:teuthology.orchestra.run.mira030.stdout:}

see


Related issues

Duplicated by RADOS - Bug #40825: test_osd_came_back (tasks.mgr.test_progress.TestProgress) ... FAIL Duplicate 07/18/2019
Duplicated by mgr - Bug #41400: Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) Duplicate 08/22/2019

History

#2 Updated by Kefu Chai 9 months ago

  • Status changed from 12 to Fix Under Review
  • Pull request ID set to 28840

#3 Updated by Sage Weil 9 months ago

  • Status changed from Fix Under Review to Resolved

#4 Updated by Kefu Chai 8 months ago

  • Duplicated by Bug #40825: test_osd_came_back (tasks.mgr.test_progress.TestProgress) ... FAIL added

#5 Updated by Kefu Chai 8 months ago

  • Status changed from Resolved to In Progress

#7 Updated by Kefu Chai 8 months ago

http://pulpito.ceph.com/kchai-2019-08-12_14:45:48-rados-wip-kefu-testing-2019-08-12-1306-distro-basic-mira/4211314/

still broken

the tested branch includes the PRs above.

2019-08-12T17:37:10.800 INFO:tasks.cephfs_test_runner:======================================================================
2019-08-12T17:37:10.800 INFO:tasks.cephfs_test_runner:ERROR: test_osd_came_back (tasks.mgr.test_progress.TestProgress)
2019-08-12T17:37:10.800 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-08-12T17:37:10.800 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-08-12T17:37:10.800 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-08-12-1306/qa/tasks/mgr/test_progress.py", line 202, in test_osd_came_back
2019-08-12T17:37:10.801 INFO:tasks.cephfs_test_runner:    ev2 = self._simulate_back_in([0], ev1)
2019-08-12T17:37:10.801 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-08-12-1306/qa/tasks/mgr/test_progress.py", line 128, in _simulate_back_in
2019-08-12T17:37:10.801 INFO:tasks.cephfs_test_runner:    timeout=self.EVENT_CREATION_PERIOD)
2019-08-12T17:37:10.801 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-08-12-1306/qa/tasks/ceph_test_case.py", line 145, in wait_until_equal
2019-08-12T17:37:10.801 INFO:tasks.cephfs_test_runner:    elapsed, expect_val, val
2019-08-12T17:37:10.801 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out after 5 seconds waiting for 1 (currently 0)
2019-08-12T17:37:10.802 INFO:tasks.cephfs_test_runner:
2019-08-12T17:37:10.802 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-08-12T17:37:10.802 INFO:tasks.cephfs_test_runner:Ran 1 test in 92.563s
2019-08-12T17:37:10.802 INFO:tasks.cephfs_test_runner:
2019-08-12T17:37:10.802 INFO:tasks.cephfs_test_runner:FAILED (errors=1)

#8 Updated by Kefu Chai 8 months ago

please note, https://github.com/ceph/ceph/pull/29625 does not address this issue.

#9 Updated by Neha Ojha 8 months ago

  • Duplicates Bug #41400: Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) added

#10 Updated by Neha Ojha 8 months ago

  • Priority changed from Normal to Urgent

#11 Updated by Kefu Chai 8 months ago

  • Duplicates deleted (Bug #41400: Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress))

#12 Updated by Kefu Chai 8 months ago

  • Duplicated by Bug #41400: Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) added

#13 Updated by Ricardo Dias 7 months ago

  • Assignee set to Ricardo Dias
  • Pull request ID changed from 28840 to 30095

Opened PR to fix QA test failures. https://github.com/ceph/ceph/pull/30095

#14 Updated by Kefu Chai 7 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF