Ceph » RADOS

/a/teuthology-2022-10-09_07:01:03-rados-quincy-distro-default-smithi/7059463

2022-10-09T17:24:06.314 INFO:teuthology.orchestra.run.smithi078.stderr:2022-10-09T17:24:06.162+0000 7fb466fb9700  1 -- 172.21.15.78:0/2058838378 >> [v2:172.21.15.88:6824/34015,v1:172.21.15.88:6825/34015] conn(0x7fb440034f40 msgr2=0x7fb4400373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2022-10-09T17:24:06.315 INFO:teuthology.orchestra.run.smithi078.stderr:2022-10-09T17:24:06.162+0000 7fb466fb9700  1 --2- 172.21.15.78:0/2058838378 >> [v2:172.21.15.88:6824/34015,v1:172.21.15.88:6825/34015] conn(0x7fb440034f40 0x7fb4400373f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb454003bd0 tx=0x7fb4540092c0 comp rx=0 tx=0).stop
2022-10-09T17:24:06.315 INFO:teuthology.orchestra.run.smithi078.stderr:2022-10-09T17:24:06.162+0000 7fb466fb9700 10 monclient: shutdown
2022-10-09T17:24:06.315 INFO:teuthology.orchestra.run.smithi078.stderr:2022-10-09T17:24:06.162+0000 7fb466fb9700 20 monclient: shutdown discarding 0 pending message(s)
2022-10-09T17:24:06.315 INFO:teuthology.orchestra.run.smithi078.stderr:2022-10-09T17:24:06.162+0000 7fb466fb9700  1 -- 172.21.15.78:0/2058838378 >> [v2:172.21.15.78:3302/0,v1:172.21.15.78:6791/0] conn(0x7fb460137660 msgr2=0x7fb46011cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2022-10-09T17:24:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2022-10-09T17:24:06.162+0000 7fb466fb9700  1 --2- 172.21.15.78:0/2058838378 >> [v2:172.21.15.78:3302/0,v1:172.21.15.78:6791/0] conn(0x7fb460137660 0x7fb46011cb00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb44800b4c0 tx=0x7fb4480090f0 comp rx=0 tx=0).stop
2022-10-09T17:24:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2022-10-09T17:24:06.162+0000 7fb465556700  1 -- 172.21.15.78:0/2058838378 reap_dead start
2022-10-09T17:24:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2022-10-09T17:24:06.162+0000 7fb465556700  1 -- 172.21.15.78:0/2058838378 reap_dead start
2022-10-09T17:24:06.316 INFO:teuthology.orchestra.run.smithi078.stderr:2022-10-09T17:24:06.162+0000 7fb466fb9700  1 -- 172.21.15.78:0/2058838378 shutdown_connections
2022-10-09T17:24:06.317 INFO:teuthology.orchestra.run.smithi078.stderr:2022-10-09T17:24:06.162+0000 7fb466fb9700  1 -- 172.21.15.78:0/2058838378 >> 172.21.15.78:0/2058838378 conn(0x7fb4600a7030 msgr2=0x7fb4600a4eb0 unknown :-1 s=STATE_NONE l=0).mark_down
2022-10-09T17:24:06.317 INFO:teuthology.orchestra.run.smithi078.stderr:2022-10-09T17:24:06.162+0000 7fb466fb9700  1 -- 172.21.15.78:0/2058838378 shutdown_connections
2022-10-09T17:24:06.317 INFO:teuthology.orchestra.run.smithi078.stderr:2022-10-09T17:24:06.162+0000 7fb466fb9700  1 -- 172.21.15.78:0/2058838378 wait complete.
2022-10-09T17:24:06.318 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status
2022-10-09T17:24:06.319 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):

Around the time of the Traceback, we can see that monitors were out of quorum:

2022-10-09T17:24:06.299+0000 7fc65e5a0700 20 mon.c@4(peon).health dump:{
    "quorum_health": {},
    "leader_health": {
        "MON_DOWN": {
            "severity": "HEALTH_WARN",
            "summary": {
                "message": "4/9 mons down, quorum a,b,c,h,e",
                "count": 4
            },
            "detail": [
                {
                    "message": "mon.f (rank 1) addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] is down (out of quorum)" 
                },
                {
                    "message": "mon.g (rank 3) addr [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] is down (out of quorum)" 
                },
                {
                    "message": "mon.d (rank 6) addr [v2:172.21.15.78:3303/0,v1:172.21.15.78:6792/0] is down (out of quorum)" 
                },
                {
                    "message": "mon.i (rank 7) addr [v2:172.21.15.88:3303/0,v1:172.21.15.88:6792/0] is down (out of quorum)" 
                }
            ]
        }
    }
}

Could be related to https://tracker.ceph.com/issues/45318, but I've seen two instances now where this fails specifically during mon/crush_ops.sh.

Second instance on Pacific:
/a/yuriw-2022-10-05_20:44:57-rados-wip-yuri4-testing-2022-10-05-0917-pacific-distro-default-smithi/7055682