Bug #52321: qa/tasks/rook times out: 'check osd count' reached maximum tries (90) after waiting for 900 seconds - Orchestrator - Ceph

Actions

Copy link

Bug #52321

open

qa/tasks/rook times out: 'check osd count' reached maximum tries (90) after waiting for 900 seconds

Added by Aishwarya Mathuria over 2 years ago. Updated over 1 year ago.

Status:

New

Priority:

High

Assignee:

Category:

mgr/rook

Target version:

% Done:

Source:

Tags:

Backport:

quincy,pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2021-08-17T08:10:34.106 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/contextutil.py", line 31, in nested
    vars.append(enter())
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/rook.py", line 489, in wait_for_osds
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds
2021-08-17T08:10:34.106 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/rook.py", line 454, in rook_toolbox
    yield
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/contextutil.py", line 31, in nested
    vars.append(enter())
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/rook.py", line 489, in wait_for_osds
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds
2021-08-17T08:10:34.107 DEBUG:teuthology.orchestra.run.smithi087:> kubectl delete -f rook/cluster/examples/kubernetes/ceph/toolbox.yaml
2021-08-17T08:10:34.184 INFO:teuthology.orchestra.run.smithi087.stdout:deployment.apps "rook-ceph-tools" deleted
2021-08-17T08:10:34.205 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/rook.py", line 391, in rook_cluster
    yield
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/contextutil.py", line 31, in nested
    vars.append(enter())
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_a00b573203353a8db1f3e59f48e827ed27479b62/qa/tasks/rook.py", line 489, in wait_for_osds
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds
2021-08-17T08:10:34.205 DEBUG:teuthology.orchestra.run.smithi087:> kubectl delete -f cluster.yaml

https://pulpito.ceph.com/yuriw-2021-08-16_21:15:00-rados-wip-yuri-testing-master-8.16.21-distro-basic-smithi/6342123

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Neha Ojha over 2 years ago

Project changed from RADOS to Orchestrator
Category set to mgr/rook

Actions

Copy link

Updated by Neha Ojha over 2 years ago

/a/yuriw-2021-08-26_18:40:53-rados-wip-yuri7-testing-2021-08-26-0841-distro-basic-smithi/6360494/

Actions

Copy link

Updated by Sebastian Wagner over 2 years ago

Description updated (diff)

Actions

Copy link

Updated by Sebastian Wagner over 2 years ago

Has duplicate Bug #52420: rados/rook: 'wait for operator' reached maximum tries (90) after waiting for 900 seconds added

Actions

Copy link

Updated by Laura Flores almost 2 years ago

Status changed from New to Pending Backport
Backport set to quincy

From Joseph S. in #52420, which was a duplicate:

Resolved, I don't remember exactly what the issue was so I can't give you a PR that solves it, but the solution is probably in one of these:
https://github.com/ceph/ceph/pull/42757
https://github.com/ceph/ceph/pull/42858
https://github.com/ceph/ceph/pull/42873
https://github.com/ceph/ceph/pull/43136

It's still happening in quincy, so I think this needs a backport.
/a/yuriw-2022-05-12_22:14:26-rados-wip-yuri-testing-2022-05-10-1027-quincy-distro-default-smithi/6831415

Actions

Copy link

Updated by Laura Flores almost 2 years ago

Status changed from Pending Backport to New

An occurrence was found in master: /a/yuriw-2022-05-19_01:43:57-rados-wip-yuri7-testing-2022-05-18-1636-distro-default-smithi/6840742

There was a note on the duplicate tracker that this issue was solved in master (see note#5), but we don't know which commit is responsible. Until that is decided, I think I will set the "Pending Backport" status back to "New" so we can verify whether the "good" commit that exists on master is helping here.

Actions

Copy link

Updated by Laura Flores almost 2 years ago

/a/yuriw-2022-05-26_23:23:48-rados-wip-yuri2-testing-2022-05-26-1430-quincy-distro-default-smithi/6849404
/a/yuriw-2022-05-26_23:23:48-rados-wip-yuri2-testing-2022-05-26-1430-quincy-distro-default-smithi/6849561
/a/yuriw-2022-05-26_23:23:48-rados-wip-yuri2-testing-2022-05-26-1430-quincy-distro-default-smithi/6849483
/a/yuriw-2022-05-26_23:23:48-rados-wip-yuri2-testing-2022-05-26-1430-quincy-distro-default-smithi/6849641

Actions

Copy link

Updated by Laura Flores almost 2 years ago

Backport changed from quincy to quincy,pacific

/a/yuriw-2022-05-31_21:35:41-rados-wip-yuri2-testing-2022-05-31-1300-pacific-distro-default-smithi/6856410

Actions

Copy link

Updated by Laura Flores almost 2 years ago

Priority changed from Normal to High

Upping the priority of this since it keeps appearing in Pacific.

Actions

Copy link

#10

Updated by Laura Flores almost 2 years ago

4 occurrences in /a/yuriw-2022-06-10_03:10:47-rados-wip-yuri4-testing-2022-06-09-1510-quincy-distro-default-smithi

Actions

Copy link

#11

Updated by Sridhar Seshasayee almost 2 years ago

/a/yuriw-2022-06-15_18:29:33-rados-wip-yuri4-testing-2022-06-15-1000-pacific-distro-default-smithi/6881248

Actions

Copy link

#12

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

/a/yuriw-2022-06-23_14:17:25-rados-wip-yuri6-testing-2022-06-22-1419-distro-default-smithi/6894623
/a/yuriw-2022-06-23_14:17:25-rados-wip-yuri6-testing-2022-06-22-1419-distro-default-smithi/6894627

Actions

Copy link

#13

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

/a/yuriw-2022-06-23_14:17:25-rados-wip-yuri6-testing-2022-06-22-1419-distro-default-smithi/6894630

Actions

Copy link

#14

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

/a/yuriw-2022-06-23_14:17:25-rados-wip-yuri6-testing-2022-06-22-1419-distro-default-smithi/6894632

Actions

Copy link

#15

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

yuriw-2022-06-30_14:20:05-rados-wip-yuri3-testing-2022-06-28-1737-distro-default-smithi/[6907398, 6907403, 6907406, 6907412]

Actions

Copy link

#16

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

/a/yuriw-2022-06-29_13:30:16-rados-wip-yuri3-testing-2022-06-28-1737-distro-default-smithi/[6905513, 6905691]

Actions

Copy link

#17

Updated by Nitzan Mordechai almost 2 years ago

/a/yuriw-2022-07-09_14:48:58-rados-wip-yuri6-testing-2022-07-07-1429-distro-default-smithi/6921591
/a/yuriw-2022-07-09_14:48:58-rados-wip-yuri6-testing-2022-07-07-1429-distro-default-smithi/6921593

Actions

Copy link

#18

Updated by Aishwarya Mathuria almost 2 years ago

/a/yuriw-2022-07-13_19:41:18-rados-wip-yuri7-testing-2022-07-11-1631-distro-default-smithi/6929398
/a/yuriw-2022-07-13_19:41:18-rados-wip-yuri7-testing-2022-07-11-1631-distro-default-smithi/6929392
/a/yuriw-2022-07-13_19:41:18-rados-wip-yuri7-testing-2022-07-11-1631-distro-default-smithi/6929402
/a/yuriw-2022-07-13_19:41:18-rados-wip-yuri7-testing-2022-07-11-1631-distro-default-smithi/6929388

Actions

Copy link

#19

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

/a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6943733

Actions

Copy link

#20

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

/a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6943896

Actions

Copy link

#21

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

/a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6944365/

Actions

Copy link

#22

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

yuriw-2022-07-24_15:38:21-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6946357
yuriw-2022-07-24_15:38:21-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6946381
yuriw-2022-07-24_15:38:21-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6946366
yuriw-2022-07-24_15:38:21-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6946373

Actions

Copy link

#23

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-04_11:58:29-rados-wip-yuri3-testing-2022-08-03-0828-pacific-distro-default-smithi/6958186

Actions

Copy link

#24

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-04_11:58:29-rados-wip-yuri3-testing-2022-08-03-0828-pacific-distro-default-smithi/6958446

Actions

Copy link

#25

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-08_22:19:17-rados-wip-yuri-testing-2022-08-08-1230-quincy-distro-default-smithi/6961818/

Actions

Copy link

#26

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-08_22:19:17-rados-wip-yuri-testing-2022-08-08-1230-quincy-distro-default-smithi/6962134/

Actions

Copy link

#27

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-08_22:19:17-rados-wip-yuri-testing-2022-08-08-1230-quincy-distro-default-smithi/6962134/

Actions

Copy link

#28

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-11_14:23:31-rados-wip-yuri-testing-2022-08-10-1103-quincy-distro-default-smithi/6967374/
/a/yuriw-2022-08-11_14:23:31-rados-wip-yuri-testing-2022-08-10-1103-quincy-distro-default-smithi/6966447

Actions

Copy link

#29

Updated by Nitzan Mordechai over 1 year ago

/a/yuriw-2022-08-16_15:48:32-rados-wip-yuri4-testing-2022-08-15-0951-distro-default-smithi/6975388

Actions

Copy link

#30

Updated by Matan Breizman over 1 year ago

/a/yuriw-2022-08-15_17:54:08-rados-wip-yuri2-testing-2022-08-15-0848-quincy-distro-default-smithi/6973678/
/a/yuriw-2022-08-15_17:54:08-rados-wip-yuri2-testing-2022-08-15-0848-quincy-distro-default-smithi/6973835/

Actions

Copy link

#31