Project

General

Profile

Actions

Bug #52321

open

qa/tasks/rook times out: 'check osd count' reached maximum tries (90) after waiting for 900 seconds

Added by Aishwarya Mathuria over 2 years ago. Updated over 1 year ago.

Status:
New
Priority:
High
Assignee:
-
Category:
mgr/rook
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-08-17T08:10:34.106 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/contextutil.py", line 31, in nested
    vars.append(enter())
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/rook.py", line 489, in wait_for_osds
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds
2021-08-17T08:10:34.106 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/rook.py", line 454, in rook_toolbox
    yield
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/contextutil.py", line 31, in nested
    vars.append(enter())
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/rook.py", line 489, in wait_for_osds
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds
2021-08-17T08:10:34.107 DEBUG:teuthology.orchestra.run.smithi087:> kubectl delete -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml
2021-08-17T08:10:34.184 INFO:teuthology.orchestra.run.smithi087.stdout:deployment.apps "rook-ceph-tools" deleted
2021-08-17T08:10:34.205 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/rook.py", line 391, in rook_cluster
    yield
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/contextutil.py", line 31, in nested
    vars.append(enter())
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/rook.py", line 489, in wait_for_osds
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds
2021-08-17T08:10:34.205 DEBUG:teuthology.orchestra.run.smithi087:> kubectl delete -f cluster.yaml

https://pulpito.ceph.com/yuriw-2021-08-16_21:15:00-rados-wip-yuri-testing-master-8.16.21-distro-basic-smithi/6342123


Related issues 1 (0 open1 closed)

Has duplicate Orchestrator - Bug #52420: rados/rook: 'wait for operator' reached maximum tries (90) after waiting for 900 secondsDuplicate

Actions
Actions #1

Updated by Neha Ojha over 2 years ago

  • Project changed from RADOS to Orchestrator
  • Category set to mgr/rook
Actions #2

Updated by Neha Ojha over 2 years ago

/a/yuriw-2021-08-26_18:40:53-rados-wip-yuri7-testing-2021-08-26-0841-distro-basic-smithi/6360494/

Actions #3

Updated by Sebastian Wagner over 2 years ago

  • Description updated (diff)
Actions #4

Updated by Sebastian Wagner over 2 years ago

  • Has duplicate Bug #52420: rados/rook: 'wait for operator' reached maximum tries (90) after waiting for 900 seconds added
Actions #5

Updated by Laura Flores almost 2 years ago

  • Status changed from New to Pending Backport
  • Backport set to quincy

From Joseph S. in #52420, which was a duplicate:

Resolved, I don't remember exactly what the issue was so I can't give you a PR that solves it, but the solution is probably in one of these:
https://github.com/ceph/ceph/pull/42757
https://github.com/ceph/ceph/pull/42858
https://github.com/ceph/ceph/pull/42873
https://github.com/ceph/ceph/pull/43136

It's still happening in quincy, so I think this needs a backport.
/a/yuriw-2022-05-12_22:14:26-rados-wip-yuri-testing-2022-05-10-1027-quincy-distro-default-smithi/6831415

Actions #6

Updated by Laura Flores almost 2 years ago

  • Status changed from Pending Backport to New

An occurrence was found in master: /a/yuriw-2022-05-19_01:43:57-rados-wip-yuri7-testing-2022-05-18-1636-distro-default-smithi/6840742

There was a note on the duplicate tracker that this issue was solved in master (see note#5), but we don't know which commit is responsible. Until that is decided, I think I will set the "Pending Backport" status back to "New" so we can verify whether the "good" commit that exists on master is helping here.

Actions #7

Updated by Laura Flores almost 2 years ago

/a/yuriw-2022-05-26_23:23:48-rados-wip-yuri2-testing-2022-05-26-1430-quincy-distro-default-smithi/6849404
/a/yuriw-2022-05-26_23:23:48-rados-wip-yuri2-testing-2022-05-26-1430-quincy-distro-default-smithi/6849561
/a/yuriw-2022-05-26_23:23:48-rados-wip-yuri2-testing-2022-05-26-1430-quincy-distro-default-smithi/6849483
/a/yuriw-2022-05-26_23:23:48-rados-wip-yuri2-testing-2022-05-26-1430-quincy-distro-default-smithi/6849641

Actions #8

Updated by Laura Flores almost 2 years ago

  • Backport changed from quincy to quincy,pacific

/a/yuriw-2022-05-31_21:35:41-rados-wip-yuri2-testing-2022-05-31-1300-pacific-distro-default-smithi/6856410

Actions #9

Updated by Laura Flores almost 2 years ago

  • Priority changed from Normal to High

Upping the priority of this since it keeps appearing in Pacific.

Actions #10

Updated by Laura Flores almost 2 years ago

4 occurrences in /a/yuriw-2022-06-10_03:10:47-rados-wip-yuri4-testing-2022-06-09-1510-quincy-distro-default-smithi

Actions #11

Updated by Sridhar Seshasayee almost 2 years ago

/a/yuriw-2022-06-15_18:29:33-rados-wip-yuri4-testing-2022-06-15-1000-pacific-distro-default-smithi/6881248

Actions #12

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

/a/yuriw-2022-06-23_14:17:25-rados-wip-yuri6-testing-2022-06-22-1419-distro-default-smithi/6894623
/a/yuriw-2022-06-23_14:17:25-rados-wip-yuri6-testing-2022-06-22-1419-distro-default-smithi/6894627

Actions #13

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

/a/yuriw-2022-06-23_14:17:25-rados-wip-yuri6-testing-2022-06-22-1419-distro-default-smithi/6894630

Actions #14

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

/a/yuriw-2022-06-23_14:17:25-rados-wip-yuri6-testing-2022-06-22-1419-distro-default-smithi/6894632

Actions #15

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

yuriw-2022-06-30_14:20:05-rados-wip-yuri3-testing-2022-06-28-1737-distro-default-smithi/[6907398, 6907403, 6907406, 6907412]

Actions #16

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

/a/yuriw-2022-06-29_13:30:16-rados-wip-yuri3-testing-2022-06-28-1737-distro-default-smithi/[6905513, 6905691]

Actions #17

Updated by Nitzan Mordechai almost 2 years ago

/a/yuriw-2022-07-09_14:48:58-rados-wip-yuri6-testing-2022-07-07-1429-distro-default-smithi/6921591
/a/yuriw-2022-07-09_14:48:58-rados-wip-yuri6-testing-2022-07-07-1429-distro-default-smithi/6921593

Actions #18

Updated by Aishwarya Mathuria almost 2 years ago

/a/yuriw-2022-07-13_19:41:18-rados-wip-yuri7-testing-2022-07-11-1631-distro-default-smithi/6929398
/a/yuriw-2022-07-13_19:41:18-rados-wip-yuri7-testing-2022-07-11-1631-distro-default-smithi/6929392
/a/yuriw-2022-07-13_19:41:18-rados-wip-yuri7-testing-2022-07-11-1631-distro-default-smithi/6929402
/a/yuriw-2022-07-13_19:41:18-rados-wip-yuri7-testing-2022-07-11-1631-distro-default-smithi/6929388

Actions #19

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6943733

Actions #20

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6943896

Actions #21

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6944365/

Actions #22

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

yuriw-2022-07-24_15:38:21-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6946357
yuriw-2022-07-24_15:38:21-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6946381
yuriw-2022-07-24_15:38:21-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6946366
yuriw-2022-07-24_15:38:21-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6946373

Actions #23

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-04_11:58:29-rados-wip-yuri3-testing-2022-08-03-0828-pacific-distro-default-smithi/6958186

Actions #24

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-04_11:58:29-rados-wip-yuri3-testing-2022-08-03-0828-pacific-distro-default-smithi/6958446

Actions #25

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-08_22:19:17-rados-wip-yuri-testing-2022-08-08-1230-quincy-distro-default-smithi/6961818/

Actions #26

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-08_22:19:17-rados-wip-yuri-testing-2022-08-08-1230-quincy-distro-default-smithi/6962134/

Actions #27

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-08_22:19:17-rados-wip-yuri-testing-2022-08-08-1230-quincy-distro-default-smithi/6962134/

Actions #28

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-11_14:23:31-rados-wip-yuri-testing-2022-08-10-1103-quincy-distro-default-smithi/6967374/
/a/yuriw-2022-08-11_14:23:31-rados-wip-yuri-testing-2022-08-10-1103-quincy-distro-default-smithi/6966447

Actions #29

Updated by Nitzan Mordechai over 1 year ago

/a/yuriw-2022-08-16_15:48:32-rados-wip-yuri4-testing-2022-08-15-0951-distro-default-smithi/6975388

Actions #30

Updated by Matan Breizman over 1 year ago

/a/yuriw-2022-08-15_17:54:08-rados-wip-yuri2-testing-2022-08-15-0848-quincy-distro-default-smithi/6973678/
/a/yuriw-2022-08-15_17:54:08-rados-wip-yuri2-testing-2022-08-15-0848-quincy-distro-default-smithi/6973835/

Actions #31

Updated by Ernesto Puerta over 1 year ago

  • Translation missing: en.field_tag_list set to test-failure
Actions #32

Updated by Laura Flores over 1 year ago

/a/yuriw-2022-08-18_17:36:35-rados-wip-yuri5-testing-2022-08-18-0812-distro-default-smithi/6978855

Actions #33

Updated by Nitzan Mordechai over 1 year ago

/a/yuriw-2022-09-01_00:21:36-rados-wip-yuri7-testing-2022-08-31-0841-distro-default-smithi/7003412

Actions #34

Updated by Matan Breizman over 1 year ago

/a/yuriw-2022-09-14_20:04:05-rados-wip-yuri3-testing-2022-09-13-0928-distro-default-smithi/7033174

/a/yuriw-2022-09-14_20:04:05-rados-wip-yuri3-testing-2022-09-13-0928-distro-default-smithi/7033178

/a/yuriw-2022-09-14_13:16:11-rados-wip-yuri6-testing-2022-09-13-1352-distro-default-smithi/7032199

/a/yuriw-2022-09-14_13:16:11-rados-wip-yuri6-testing-2022-09-13-1352-distro-default-smithi/7032359

Actions #35

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

yuriw-2022-09-27_23:37:28-rados-wip-yuri2-testing-2022-09-27-1455-distro-default-smithi/7046227

Actions #36

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-10-12_16:24:50-rados-wip-yuri8-testing-2022-10-12-0718-quincy-distro-default-smithi/7063706/

Actions #37

Updated by Matan Breizman over 1 year ago

/a/yuriw-2022-11-28_21:13:47-rados-wip-yuri11-testing-2022-11-18-1506-distro-default-smithi/7095046/

Actions

Also available in: Atom PDF