Ceph » Messengers

the mark_down comes from mark_down_all() in bootstrap(), which is possibly no longer necessary, since the msgr tries reasonably hard not to identify itself beyond the type.

  if (newrank != rank) {
    dout(0) << " my rank is now " << newrank << " (was " << rank << ")" << dendl;
    messenger->set_myname(entity_name_t::MON(newrank));
    rank = newrank;

    // reset all connections, or else our peers will think we are someone else.
    messenger->mark_down_all();
  }

So the quorum forms correctly, but just there is a monitor that doesn't get in because it's not connecting to anybody?

another more recent instance

2019-01-10 02:29:57.299 7f62ea13e700 20 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] 0x4a7ac00 conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_1 r=0
2019-01-10 02:29:57.299 7f62ea13e700 20 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] 0x4a7ac00 conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).wait_connect_message_auth
2019-01-10 02:29:57.299 7f62ea13e700 20 -- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=158
2019-01-10 02:29:57.299 7f62ea13e700 20 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] 0x4a7ac00 conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_auth r=0
2019-01-10 02:29:57.299 7f62ea13e700 20 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] 0x4a7ac00 conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2
2019-01-10 02:29:57.299 7f62ea13e700 20 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] 0x4a7ac00 conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept got peer connect_seq 0 global_seq 14
2019-01-10 02:29:57.299 7f62ea13e700 10 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] 0x4a7ac00 conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept of host_type 1, policy.lossy=0 policy.server=0 policy.standby=1 policy.resetcheck=1
2019-01-10 02:29:57.299 7f62ea13e700 10 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] 0x4a7ac00 conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept my proto 13, their proto 13
2019-01-10 02:29:57.299 7f62ea13e700 10 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] 0x4a7ac00 conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 authorizor_protocol 2 len 158
2019-01-10 02:29:57.299 7f62ea13e700 10 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] 0x4a7ac00 conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept setting up session_security.
2019-01-10 02:29:57.299 7f6302e9df00 20 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] 0x4a7ac00 conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).stop
2019-01-10 02:29:57.299 7f6302e9df00  2 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] 0x4a7ac00 conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).stop
2019-01-10 02:29:57.299 7f6302e9df00 10 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3306/0,v1:172.21.15.150:6795/0] 0x4a7ac00 conn(0x4b6fa80 msgr2=0x4a7ac00 :36622 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).discard_out_queue started
2019-01-10 02:29:57.299 7f6302e9df00  5 -- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] shutdown_connections delete 0x4b6f600
2019-01-10 02:29:57.299 7f6302e9df00  5 -- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] shutdown_connections delete 0x4b6fa80
2019-01-10 02:29:57.299 7f62e993d700 10 -- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] accept_conn 0x4b6f600 [v2:172.21.15.150:3304/0,v1:172.21.15.150:6793/0]
2019-01-10 02:29:57.299 7f62e993d700  1 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3304/0,v1:172.21.15.150:6793/0] 0x4a7a600 conn(0x4b6f600 msgr2=0x4a7a600 :36618 s=CLOSED pgs=12 cs=1 l=0).open state changed while accept_conn, it must be mark_down
2019-01-10 02:29:57.299 7f62e993d700 10 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3304/0,v1:172.21.15.150:6793/0] 0x4a7a600 conn(0x4b6f600 msgr2=0x4a7a600 :36618 s=CLOSED pgs=12 cs=1 l=0).accept fault after register
2019-01-10 02:29:57.299 7f62e993d700 20 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3304/0,v1:172.21.15.150:6793/0] 0x4a7a600 conn(0x4b6f600 msgr2=0x4a7a600 :36618 s=CLOSED pgs=12 cs=1 l=0).fault
2019-01-10 02:29:57.299 7f62e993d700 10 --2- [v2:172.21.15.150:3303/0,v1:172.21.15.150:6792/0] >> [v2:172.21.15.150:3304/0,v1:172.21.15.150:6793/0] 0x4a7a600 conn(0x4b6f600 msgr2=0x4a7a600 :36618 s=CLOSED pgs=12 cs=1 l=0).fault connection is already closed

just a note, i was able to reliably reproduce this with rados/multimon (no subsets), filtering to only do the 21.yaml, mon_recovery.yaml, and async messenger.. usually 2-3 failures per run (of 92 tests), with the symptom that the quorum_status command timed out.