Project

General

Profile

Actions

Bug #54029

closed

orch:cephadm/workunits/{agent/on mon_election/connectivity task/test_orch_cli} test failing

Added by Adam King about 2 years ago. Updated 12 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This specific test is failing due to the cpehfs-mirror daemon not appearing up in "ceph orch ps" output within 30 seconds of applying it. Seen this on a few different runs over the last 24 hours, two of which were testing a ceph-volume change that is entirely unrelated.

https://pulpito.ceph.com/adking-2022-01-26_05:20:46-orch:cephadm-wip-adk-testing-2022-01-25-2134-distro-basic-smithi/6641250/
https://pulpito.ceph.com/gabrioux-2022-01-26_10:55:38-rados:cephadm-wip-guits-testing-2022-01-26-0910-distro-default-smithi/6641319/
https://pulpito.ceph.com/gabrioux-2022-01-26_13:07:05-orch:cephadm-wip-guits-testing-2022-01-26-0910-distro-default-smithi/6641423

2022-01-26T07:15:36.658 INFO:tasks.cephfs_test_runner:test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI) ... ERROR
2022-01-26T07:15:36.658 INFO:tasks.cephfs_test_runner:
2022-01-26T07:15:36.659 INFO:tasks.cephfs_test_runner:======================================================================
2022-01-26T07:15:36.659 INFO:tasks.cephfs_test_runner:ERROR: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI)
2022-01-26T07:15:36.659 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-01-26T07:15:36.659 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-01-26T07:15:36.660 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_ad9722f2c71c5a0115cf70ffd5040e21f3fa73ee/qa/tasks/cephadm_cases/test_cli.py", line 70, in test_cephfs_mirror
2022-01-26T07:15:36.660 INFO:tasks.cephfs_test_runner:    self.wait_until_true(lambda: 'cephfs-mirror' in self._orch_cmd('ps'), 30)
2022-01-26T07:15:36.660 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_ad9722f2c71c5a0115cf70ffd5040e21f3fa73ee/qa/tasks/ceph_test_case.py", line 212, in wait_until_true
2022-01-26T07:15:36.661 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2022-01-26T07:15:36.661 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries
2022-01-26T07:15:36.661 INFO:tasks.cephfs_test_runner:
2022-01-26T07:15:36.661 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-01-26T07:15:36.662 INFO:tasks.cephfs_test_runner:Ran 1 test in 63.676s
2022-01-26T07:15:36.662 INFO:tasks.cephfs_test_runner:
2022-01-26T07:15:36.662 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2022-01-26T07:15:36.663 INFO:tasks.cephfs_test_runner:
2022-01-26T07:15:36.663 INFO:tasks.cephfs_test_runner:======================================================================
2022-01-26T07:15:36.663 INFO:tasks.cephfs_test_runner:ERROR: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI)
2022-01-26T07:15:36.664 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-01-26T07:15:36.664 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-01-26T07:15:36.664 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_ad9722f2c71c5a0115cf70ffd5040e21f3fa73ee/qa/tasks/cephadm_cases/test_cli.py", line 70, in test_cephfs_mirror
2022-01-26T07:15:36.664 INFO:tasks.cephfs_test_runner:    self.wait_until_true(lambda: 'cephfs-mirror' in self._orch_cmd('ps'), 30)
2022-01-26T07:15:36.665 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_ad9722f2c71c5a0115cf70ffd5040e21f3fa73ee/qa/tasks/ceph_test_case.py", line 212, in wait_until_true
2022-01-26T07:15:36.665 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2022-01-26T07:15:36.665 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries

Related issues 1 (1 open0 closed)

Copied to Orchestrator - Bug #57755: task/test_orch_cli: test_cephfs_mirror times outNew

Actions
Actions #1

Updated by Laura Flores about 2 years ago

/a/benhanokh-2022-01-26_21:12:05-rados-WIP_GBH_NCB_new_alloc_map_A6-distro-basic-smithi/6642205
Description: rados/cephadm/workunits/{agent/on mon_election/classic task/test_orch_cli}

Actions #2

Updated by Neha Ojha about 2 years ago

/a/yuriw-2022-01-27_15:12:15-rados-wip-yuri7-testing-2022-01-26-1547-distro-default-smithi/6644411

Actions #3

Updated by Laura Flores about 2 years ago

/a/yuriw-2022-02-11_22:59:19-rados-wip-yuri4-testing-2022-02-11-0858-distro-default-smithi/6677979

Actions #4

Updated by Adam King about 2 years ago

  • Priority changed from Normal to High
  • Pull request ID set to 44965

merged in the now linked PR and all the test_orch_cli tests passed in http://pulpito.front.sepia.ceph.com/adking-2022-02-11_16:35:44-orch:cephadm-wip-adk-testing-2022-02-10-1637-distro-basic-smithi/. If no more failures are reported in the next few days I'll mark this resolved

Actions #5

Updated by Adam King about 2 years ago

  • Status changed from New to Fix Under Review
Actions #6

Updated by Laura Flores about 2 years ago

Another failure, but in Quincy:
/a/yuriw-2022-02-28_21:23:00-rados-wip-yuri-testing-2022-02-28-0823-quincy-distro-default-smithi/6712433

Actions #7

Updated by Adam King about 2 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to quincy
Actions #8

Updated by Laura Flores about 2 years ago

/a/yuriw-2022-03-24_01:58:56-rados-wip-yuri7-testing-2022-03-23-1332-quincy-distro-default-smithi/6756818

Actions #9

Updated by Laura Flores about 2 years ago

Actions #10

Updated by Aishwarya Mathuria about 2 years ago

/a/yuriw-2022-03-29_21:35:32-rados-wip-yuri5-testing-2022-03-29-1152-quincy-distro-default-smithi/6767916

Actions #11

Updated by Aishwarya Mathuria about 2 years ago

/a/yuriw-2022-03-31_21:45:19-rados-wip-yuri5-testing-2022-03-31-1158-quincy-distro-default-smithi/6770539

Actions #12

Updated by Laura Flores about 2 years ago

/a/yuriw-2022-03-31_21:46:00-rados-wip-yuri4-testing-2022-03-31-1158-quincy-distro-default-smithi/6770616

Actions #13

Updated by Yuri Weinstein almost 2 years ago

Laura Flores wrote:

Backport PR for quincy: https://github.com/ceph/ceph/pull/45625

merged

Actions #14

Updated by Redouane Kachach Elhichou almost 2 years ago

  • Backport changed from quincy to quincy,pacific
Actions #15

Updated by Redouane Kachach Elhichou almost 2 years ago

  • Status changed from Pending Backport to Resolved
Actions #16

Updated by Matan Breizman over 1 year ago

Failing with new timeout as well. Should we open a new tracker?

2022-09-16T05:00:33.816 INFO:tasks.cephfs_test_runner:======================================================================
2022-09-16T05:00:33.817 INFO:tasks.cephfs_test_runner:ERROR: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI)
2022-09-16T05:00:33.817 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-09-16T05:00:33.818 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-09-16T05:00:33.818 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_b5f1ad94e19b4d646020c0622d3e1dd8df5b673f/qa/tasks/cephadm_cases/test_cli.py", line 70, in test_cephfs_mirror
2022-09-16T05:00:33.818 INFO:tasks.cephfs_test_runner:    self.wait_until_true(lambda: 'cephfs-mirror' in self._orch_cmd('ps'), 60)
2022-09-16T05:00:33.819 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_b5f1ad94e19b4d646020c0622d3e1dd8df5b673f/qa/tasks/ceph_test_case.py", line 212, in wait_until_true
2022-09-16T05:00:33.819 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2022-09-16T05:00:33.819 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 60s and 0 retries
2022-09-16T05:00:33.820 INFO:tasks.cephfs_test_runner:
2022-09-16T05:00:33.820 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-09-16T05:00:33.820 INFO:tasks.cephfs_test_runner:Ran 1 test in 97.566s
2022-09-16T05:00:33.821 INFO:tasks.cephfs_test_runner:
2022-09-16T05:00:33.821 INFO:tasks.cephfs_test_runner:FAILED (errors=1)

/a/yuriw-2022-09-14_13:16:11-rados-wip-yuri6-testing-2022-09-13-1352-distro-default-smithi/7032334

Actions #17

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

yuriw-2022-09-27_23:37:28-rados-wip-yuri2-testing-2022-09-27-1455-distro-default-smithi/7046362

Actions #18

Updated by Laura Flores over 1 year ago

@Matan Breizman yes, let's create a new Tracker. I found the same issue on a test branch which includes the patch for this Tracker.

Actions #19

Updated by Laura Flores over 1 year ago

  • Copied to Bug #57755: task/test_orch_cli: test_cephfs_mirror times out added
Actions #20

Updated by Laura Flores 12 months ago

  • Translation missing: en.field_tag_list set to test-failure
Actions

Also available in: Atom PDF