Bug #64968: mon: MON_DOWN warnings when mons are first booting - RADOS - Ceph

Actions

Copy link

Bug #64968

open

mon: MON_DOWN warnings when mons are first booting

Added by Patrick Donnelly about 2 months ago. Updated about 1 month ago.

Status:

Pending Backport

Priority:

Normal

Assignee:

Patrick Donnelly

Category:

Administration/Usability

Target version:

Ceph - v20.0.0

% Done:

Source:

Q/A

Tags:

backport_processed

Backport:

squid,reef

Regression:

Severity:

4 - irritation

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Monitor

Pull request ID:

56271

Crash signature (v1):

Crash signature (v2):

Description

2024-03-18T00:42:36.045 INFO:teuthology.orchestra.run.smithi162.stdout:Creating initial monmap...
2024-03-18T00:42:36.935 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap
2024-03-18T00:42:36.935 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy
2024-03-18T00:42:36.935 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1b79961c-e4c0-11ee-95c9-87774f69a715
2024-03-18T00:42:36.935 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
2024-03-18T00:42:36.935 INFO:teuthology.orchestra.run.smithi162.stdout:monmaptool for a [v2:172.21.15.162:3300,v1:172.21.15.162:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap
2024-03-18T00:42:36.936 INFO:teuthology.orchestra.run.smithi162.stdout:setting min_mon_release = quincy
2024-03-18T00:42:36.936 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: set fsid to 1b79961c-e4c0-11ee-95c9-87774f69a715
2024-03-18T00:42:36.936 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
2024-03-18T00:42:36.936 INFO:teuthology.orchestra.run.smithi162.stdout:
...
2024-03-18T00:42:36.936 INFO:teuthology.orchestra.run.smithi162.stdout:Creating mon...
2024-03-18T00:42:37.841 INFO:teuthology.orchestra.run.smithi162.stdout:create mon.a on
...
2024-03-18T00:44:20.339 INFO:tasks.cephadm:Adding mon.a on smithi162
2024-03-18T00:44:20.339 INFO:tasks.cephadm:Adding mon.b on smithi179
2024-03-18T00:44:20.339 INFO:tasks.cephadm:Adding mon.c on smithi181
...
2024-03-18T00:44:23.616 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:23 smithi162 ceph-mon[32185]: Deploying daemon mon.c on smithi181
...
2024-03-18T00:44:33.354 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 18 00:44:32 smithi181 ceph-mon[35664]: Deploying daemon mon.b on smithi179
2024-03-18T00:44:33.354 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 18 00:44:32 smithi181 ceph-mon[35664]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail
2024-03-18T00:44:33.354 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 18 00:44:32 smithi181 ceph-mon[35664]: from='mgr.14152 172.21.15.162:0/851257622' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch
2024-03-18T00:44:33.365 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:32 smithi162 ceph-mon[32185]: from='mgr.14152 172.21.15.162:0/851257622' entity='mgr.x'
2024-03-18T00:44:33.365 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:32 smithi162 ceph-mon[32185]: from='mgr.14152 172.21.15.162:0/851257622' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch
2024-03-18T00:44:33.365 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:32 smithi162 ceph-mon[32185]: from='mgr.14152 172.21.15.162:0/851257622' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
2024-03-18T00:44:33.365 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:32 smithi162 ceph-mon[32185]: Deploying daemon mon.b on smithi179
...
2024-03-18T00:44:41.117 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:40 smithi162 ceph-mon[32185]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)
2024-03-18T00:44:41.117 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:40 smithi162 ceph-mon[32185]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c
2024-03-18T00:44:41.117 INFO:journalctl@ceph.mon.a.smithi162.stdout:Mar 18 00:44:40 smithi162 ceph-mon[32185]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c

From: /teuthology/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608542/teuthology.log

Related issues 2 (1 open — 1 closed)