Project

General

Profile

Bug #54029

orch:cephadm/workunits/{agent/on mon_election/connectivity task/test_orch_cli} test failing

Added by Adam King about 2 years ago. Updated 11 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This specific test is failing due to the cpehfs-mirror daemon not appearing up in "ceph orch ps" output within 30 seconds of applying it. Seen this on a few different runs over the last 24 hours, two of which were testing a ceph-volume change that is entirely unrelated.

https://pulpito.ceph.com/adking-2022-01-26_05:20:46-orch:cephadm-wip-adk-testing-2022-01-25-2134-distro-basic-smithi/6641250/
https://pulpito.ceph.com/gabrioux-2022-01-26_10:55:38-rados:cephadm-wip-guits-testing-2022-01-26-0910-distro-default-smithi/6641319/
https://pulpito.ceph.com/gabrioux-2022-01-26_13:07:05-orch:cephadm-wip-guits-testing-2022-01-26-0910-distro-default-smithi/6641423

2022-01-26T07:15:36.658 INFO:tasks.cephfs_test_runner:test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI) ... ERROR
2022-01-26T07:15:36.658 INFO:tasks.cephfs_test_runner:
2022-01-26T07:15:36.659 INFO:tasks.cephfs_test_runner:======================================================================
2022-01-26T07:15:36.659 INFO:tasks.cephfs_test_runner:ERROR: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI)
2022-01-26T07:15:36.659 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-01-26T07:15:36.659 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-01-26T07:15:36.660 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_ad9722f2c71c5a0115cf70ffd5040e21f3fa73ee/qa/tasks/cephadm_cases/test_cli.py", line 70, in test_cephfs_mirror
2022-01-26T07:15:36.660 INFO:tasks.cephfs_test_runner:    self.wait_until_true(lambda: 'cephfs-mirror' in self._orch_cmd('ps'), 30)
2022-01-26T07:15:36.660 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_ad9722f2c71c5a0115cf70ffd5040e21f3fa73ee/qa/tasks/ceph_test_case.py", line 212, in wait_until_true
2022-01-26T07:15:36.661 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2022-01-26T07:15:36.661 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries
2022-01-26T07:15:36.661 INFO:tasks.cephfs_test_runner:
2022-01-26T07:15:36.661 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-01-26T07:15:36.662 INFO:tasks.cephfs_test_runner:Ran 1 test in 63.676s
2022-01-26T07:15:36.662 INFO:tasks.cephfs_test_runner:
2022-01-26T07:15:36.662 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2022-01-26T07:15:36.663 INFO:tasks.cephfs_test_runner:
2022-01-26T07:15:36.663 INFO:tasks.cephfs_test_runner:======================================================================
2022-01-26T07:15:36.663 INFO:tasks.cephfs_test_runner:ERROR: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI)
2022-01-26T07:15:36.664 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-01-26T07:15:36.664 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-01-26T07:15:36.664 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_ad9722f2c71c5a0115cf70ffd5040e21f3fa73ee/qa/tasks/cephadm_cases/test_cli.py", line 70, in test_cephfs_mirror
2022-01-26T07:15:36.664 INFO:tasks.cephfs_test_runner:    self.wait_until_true(lambda: 'cephfs-mirror' in self._orch_cmd('ps'), 30)
2022-01-26T07:15:36.665 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_ad9722f2c71c5a0115cf70ffd5040e21f3fa73ee/qa/tasks/ceph_test_case.py", line 212, in wait_until_true
2022-01-26T07:15:36.665 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2022-01-26T07:15:36.665 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries

Related issues

Copied to Orchestrator - Bug #57755: task/test_orch_cli: test_cephfs_mirror times out New

History

#1 Updated by Laura Flores about 2 years ago

/a/benhanokh-2022-01-26_21:12:05-rados-WIP_GBH_NCB_new_alloc_map_A6-distro-basic-smithi/6642205
Description: rados/cephadm/workunits/{agent/on mon_election/classic task/test_orch_cli}

#2 Updated by Neha Ojha about 2 years ago

/a/yuriw-2022-01-27_15:12:15-rados-wip-yuri7-testing-2022-01-26-1547-distro-default-smithi/6644411

#3 Updated by Laura Flores about 2 years ago

/a/yuriw-2022-02-11_22:59:19-rados-wip-yuri4-testing-2022-02-11-0858-distro-default-smithi/6677979

#4 Updated by Adam King about 2 years ago

  • Priority changed from Normal to High
  • Pull request ID set to 44965

merged in the now linked PR and all the test_orch_cli tests passed in http://pulpito.front.sepia.ceph.com/adking-2022-02-11_16:35:44-orch:cephadm-wip-adk-testing-2022-02-10-1637-distro-basic-smithi/. If no more failures are reported in the next few days I'll mark this resolved

#5 Updated by Adam King about 2 years ago

  • Status changed from New to Fix Under Review

#6 Updated by Laura Flores about 2 years ago

Another failure, but in Quincy:
/a/yuriw-2022-02-28_21:23:00-rados-wip-yuri-testing-2022-02-28-0823-quincy-distro-default-smithi/6712433

#7 Updated by Adam King about 2 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to quincy

#8 Updated by Laura Flores about 2 years ago

/a/yuriw-2022-03-24_01:58:56-rados-wip-yuri7-testing-2022-03-23-1332-quincy-distro-default-smithi/6756818

#9 Updated by Laura Flores about 2 years ago

#10 Updated by Aishwarya Mathuria almost 2 years ago

/a/yuriw-2022-03-29_21:35:32-rados-wip-yuri5-testing-2022-03-29-1152-quincy-distro-default-smithi/6767916

#11 Updated by Aishwarya Mathuria almost 2 years ago

/a/yuriw-2022-03-31_21:45:19-rados-wip-yuri5-testing-2022-03-31-1158-quincy-distro-default-smithi/6770539

#12 Updated by Laura Flores almost 2 years ago

/a/yuriw-2022-03-31_21:46:00-rados-wip-yuri4-testing-2022-03-31-1158-quincy-distro-default-smithi/6770616

#13 Updated by Yuri Weinstein almost 2 years ago

Laura Flores wrote:

Backport PR for quincy: https://github.com/ceph/ceph/pull/45625

merged

#14 Updated by Redouane Kachach Elhichou almost 2 years ago

  • Backport changed from quincy to quincy,pacific

#15 Updated by Redouane Kachach Elhichou almost 2 years ago

  • Status changed from Pending Backport to Resolved

#16 Updated by Matan Breizman over 1 year ago

Failing with new timeout as well. Should we open a new tracker?

2022-09-16T05:00:33.816 INFO:tasks.cephfs_test_runner:======================================================================
2022-09-16T05:00:33.817 INFO:tasks.cephfs_test_runner:ERROR: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI)
2022-09-16T05:00:33.817 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-09-16T05:00:33.818 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-09-16T05:00:33.818 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_b5f1ad94e19b4d646020c0622d3e1dd8df5b673f/qa/tasks/cephadm_cases/test_cli.py", line 70, in test_cephfs_mirror
2022-09-16T05:00:33.818 INFO:tasks.cephfs_test_runner:    self.wait_until_true(lambda: 'cephfs-mirror' in self._orch_cmd('ps'), 60)
2022-09-16T05:00:33.819 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_b5f1ad94e19b4d646020c0622d3e1dd8df5b673f/qa/tasks/ceph_test_case.py", line 212, in wait_until_true
2022-09-16T05:00:33.819 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2022-09-16T05:00:33.819 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 60s and 0 retries
2022-09-16T05:00:33.820 INFO:tasks.cephfs_test_runner:
2022-09-16T05:00:33.820 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-09-16T05:00:33.820 INFO:tasks.cephfs_test_runner:Ran 1 test in 97.566s
2022-09-16T05:00:33.821 INFO:tasks.cephfs_test_runner:
2022-09-16T05:00:33.821 INFO:tasks.cephfs_test_runner:FAILED (errors=1)

/a/yuriw-2022-09-14_13:16:11-rados-wip-yuri6-testing-2022-09-13-1352-distro-default-smithi/7032334

#17 Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

yuriw-2022-09-27_23:37:28-rados-wip-yuri2-testing-2022-09-27-1455-distro-default-smithi/7046362

#18 Updated by Laura Flores over 1 year ago

@Matan yes, let's create a new Tracker. I found the same issue on a test branch which includes the patch for this Tracker.

#19 Updated by Laura Flores over 1 year ago

  • Copied to Bug #57755: task/test_orch_cli: test_cephfs_mirror times out added

#20 Updated by Laura Flores 11 months ago

  • Tags set to test-failure

Also available in: Atom PDF