Project

General

Profile

Actions

Bug #62535

open

cephadm: wait for healthy state times out because cephadm agent is down

Added by Laura Flores 9 months ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2023-08-17_21:18:20-rados-wip-yuri11-testing-2023-08-17-0823-distro-default-smithi/7372266

2023-08-18T00:06:10.823 INFO:teuthology.orchestra.run.smithi033.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_AGENT_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 Cephadm Agent(s) are not reporting. Hosts may be offline","count":1},"muted":false}},"mutes":[]}
2023-08-18T00:06:11.350 INFO:journalctl@ceph.mon.c.smithi033.stdout:Aug 18 00:06:11 smithi033 bash[31090]: audit 2023-08-18T00:06:10.819692+0000 mon.a (mon.0) 1235 : audit 0 from='client.? 172.21.15.33:0/2093029190' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch
2023-08-18T00:06:11.351 INFO:journalctl@ceph.mon.a.smithi033.stdout:Aug 18 00:06:11 smithi033 bash[24006]: audit 2023-08-18T00:06:10.819692+0000 mon.a (mon.0) 1235 : audit 0 from='client.? 172.21.15.33:0/2093029190' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch
2023-08-18T00:06:11.402 INFO:journalctl@ceph.mon.b.smithi089.stdout:Aug 18 00:06:11 smithi089 bash[24804]: audit 2023-08-18T00:06:10.819692+0000 mon.a (mon.0) 1235 : audit 0 from='client.? 172.21.15.33:0/2093029190' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch
2023-08-18T00:06:11.668 INFO:tasks.cephadm:Teardown begin
2023-08-18T00:06:11.669 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 32, in nested
    yield vars
  File "/home/teuthworker/src/github.com_ceph_ceph-c_251cb76b0794f2fb1d0885e3e4cf5075353ee7a0/qa/tasks/cephadm.py", line 1753, in task
    healthy(ctx=ctx, config=config)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_251cb76b0794f2fb1d0885e3e4cf5075353ee7a0/qa/tasks/ceph.py", line 1500, in healthy
    manager.wait_until_healthy(timeout=300)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_251cb76b0794f2fb1d0885e3e4cf5075353ee7a0/qa/tasks/ceph_manager.py", line 3169, in wait_until_healthy
    assert time.time() - start < timeout, \
AssertionError: timeout expired in wait_until_healthy

Actions #1

Updated by Nitzan Mordechai 6 months ago

/a/yuriw-2023-10-30_15:34:36-rados-wip-yuri10-testing-2023-10-27-0804-distro-default-smithi/7441374

Actions #2

Updated by Laura Flores 6 months ago

/a/yuriw-2023-10-31_14:43:48-rados-wip-yuri4-testing-2023-10-30-1117-distro-default-smithi/7442279

Actions #3

Updated by Laura Flores about 1 month ago

/a/yuriw-2024-03-20_18:33:32-rados-wip-yuri6-testing-2024-03-18-1406-squid-distro-default-smithi/7613354

Actions

Also available in: Atom PDF