Ceph » Orchestrator

/a/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629110

2024-03-29T05:01:11.549 INFO:teuthology.orchestra.run.smithi037.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]}
2024-03-29T05:01:12.276 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa/tasks/ceph.py", line 1478, in healthy
    manager.wait_until_healthy(timeout=300)
  File "/home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy
    assert time.time() - start < timeout, \
AssertionError: timeout expired in wait_until_healthy
2024-03-29T05:01:12.515 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e59cb050fb0241f288e174f675f76ff9
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa/tasks/ceph.py", line 1478, in healthy
    manager.wait_until_healthy(timeout=300)
  File "/home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy
    assert time.time() - start < timeout, \
AssertionError: timeout expired in wait_until_healthy

/a/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629110/remote/smithi037/log/7d9647e2-ed84-11ee-95d0-87774f69a715/ceph-mon.a.log.gz

2024-03-29T04:47:18.893+0000 7f240fb4a700  0 log_channel(cluster) log [WRN] : Health detail: HEALTH_WARN 1 failed cephadm daemon(s)
2024-03-29T04:47:18.893+0000 7f240fb4a700  1 -- [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- log(1 entries from seq 15 at 2024-03-29T04:47:18.896538+0000) -- 0x55d32dbc8a00 con 0x55d32cc06c00
2024-03-29T04:47:18.893+0000 7f240fb4a700  0 log_channel(cluster) log [WRN] : [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)
2024-03-29T04:47:18.893+0000 7f240fb4a700  1 -- [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- log(1 entries from seq 16 at 2024-03-29T04:47:18.896567+0000) -- 0x55d32dbc8c00 con 0x55d32cc06c00
2024-03-29T04:47:18.893+0000 7f240fb4a700  0 log_channel(cluster) log [WRN] :     daemon grafana.a on smithi193 is in unknown state
2024-03-29T04:47:18.893+0000 7f240fb4a700  1 -- [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- log(1 entries from seq 17 at 2024-03-29T04:47:18.896577+0000) -- 0x55d32dbc8e00 con 0x55d32cc06c00

Possibly relevent from the mgr log:
/a/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629110/remote/smithi037/log/7d9647e2-ed84-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz

2024-03-29T04:43:38.913+0000 7f5b14aae700  0 [cephadm DEBUG cephadm.serve] err: Non-zero exit code 1 from /usr/bin/docker inspect --format {{.ID}},{{.RepoDigests}} quay.ceph.io/ceph-ci/ceph@sha256:a6b710b62a1696018e5d2e3dd0dc6f55c1b3bad43842a897f9227eec10772f28
/usr/bin/docker: stdout
/usr/bin/docker: stderr Error: No such object: quay.ceph.io/ceph-ci/ceph@sha256:a6b710b62a1696018e5d2e3dd0dc6f55c1b3bad43842a897f9227eec10772f28
Traceback (most recent call last):
  File "/var/lib/ceph/7d9647e2-ed84-11ee-95d0-87774f69a715/cephadm.050dad7a7fd907bc9e41593f8b2dd060ab55f056a845bd23873115b31226ca05", line 9843, in <module>
    main()
  File "/var/lib/ceph/7d9647e2-ed84-11ee-95d0-87774f69a715/cephadm.050dad7a7fd907bc9e41593f8b2dd060ab55f056a845bd23873115b31226ca05", line 9831, in main
    r = ctx.func(ctx)
  File "/var/lib/ceph/7d9647e2-ed84-11ee-95d0-87774f69a715/cephadm.050dad7a7fd907bc9e41593f8b2dd060ab55f056a845bd23873115b31226ca05", line 2204, in _infer_image
    return func(ctx)
  File "/var/lib/ceph/7d9647e2-ed84-11ee-95d0-87774f69a715/cephadm.050dad7a7fd907bc9e41593f8b2dd060ab55f056a845bd23873115b31226ca05", line 4805, in command_inspect_image
    out, err, ret = call_throws(ctx, [
  File "/var/lib/ceph/7d9647e2-ed84-11ee-95d0-87774f69a715/cephadm.050dad7a7fd907bc9e41593f8b2dd060ab55f056a845bd23873115b31226ca05", line 1866, in call_throws
    raise RuntimeError(f'Failed command: {" ".join(command)}: {s}')
RuntimeError: Failed command: /usr/bin/docker inspect --format {{.ID}},{{.RepoDigests}} quay.ceph.io/ceph-ci/ceph@sha256:a6b710b62a1696018e5d2e3dd0dc6f55c1b3bad43842a897f9227eec10772f28: Error: No such object: quay.ceph.io/ceph-ci/ceph@sha256:a6b710b62a1696018e5d2e3dd0dc6f55c1b3bad43842a897f9227eec10772f28

This seems most relevant. Failed to parse the grafana.ini file due to permission problems:

./remote/smithi193/syslog/misc.log.gz:2024-03-29T04:45:59.553261+00:00 smithi193 systemd[1]: Started Ceph grafana.a for 7d9647e2-ed84-11ee-95d0-87774f69a715.
./remote/smithi193/syslog/misc.log.gz:2024-03-29T04:46:00.597804+00:00 smithi193 bash[23132]: audit 2024-03-29T04:45:59.645716+0000 mon.a (mon.0) 1072 : audit [DBG] from='mgr.24595 172.21.15.193:0/3288050078' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch
./remote/smithi193/syslog/misc.log.gz:2024-03-29T04:46:00.597843+00:00 smithi193 bash[23132]: audit 2024-03-29T04:45:59.649008+0000 mgr.x (mgr.24595) 56 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch
./remote/smithi193/syslog/misc.log.gz:2024-03-29T04:46:00.778482+00:00 smithi193 bash[58732]: logger=settings t=2024-03-29T04:46:00.777700702Z level=error msg="failed to parse \"/etc/grafana/grafana.ini\": open /etc/grafana/grafana.ini: permission denied" 
./remote/smithi193/syslog/misc.log.gz:2024-03-29T04:46:01.478768+00:00 smithi193 systemd[1]: ceph-7d9647e2-ed84-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE
./remote/smithi193/syslog/misc.log.gz:2024-03-29T04:46:01.525259+00:00 smithi193 systemd[1]: ceph-7d9647e2-ed84-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'.
./remote/smithi193/syslog/misc.log.gz:2024-03-29T04:46:11.680456+00:00 smithi193 systemd[1]: ceph-7d9647e2-ed84-11ee-95d0-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1.