-919> 2019-01-07 21:14:20.108 7f6343646700 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).wait_connect_message_auth
-918> 2019-01-07 21:14:20.108 7f6343646700 20 -- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=158
-914> 2019-01-07 21:14:20.108 7f6343646700 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_auth r=0
-913> 2019-01-07 21:14:20.108 7f6343646700 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2
-912> 2019-01-07 21:14:20.108 7f6343646700 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept got peer connect_seq 0 global_seq 17
-911> 2019-01-07 21:14:20.108 7f6343646700 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept of host_type 1, policy.lossy=0 policy.server=0 poli
cy.standby=1 policy.resetcheck=1
-910> 2019-01-07 21:14:20.108 7f6343646700 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept my proto 13, their proto 13
-909> 2019-01-07 21:14:20.108 7f6343646700 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 authorizor_protocol 2 len 158
-908> 2019-01-07 21:14:20.108 7f635cba6f00 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).stop
-907> 2019-01-07 21:14:20.108 7f635cba6f00 2 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).stop
-906> 2019-01-07 21:14:20.108 7f635cba6f00 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).discard_out_queue started
-903> 2019-01-07 21:14:20.108 7f634664c700 10 mon.g@7(probing) e0 ms_handle_reset 0x3719a80 v2:172.21.15.181:3303/0
-884> 2019-01-07 21:14:20.108 7f635cba6f00 5 -- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] shutdown_connections delete 0x3719a80
-559> 2019-01-07 21:14:20.116 7f6343646700 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=CLOSED pgs=0 cs=0 l=0).handle_connect_message_2: challenging authorizer
-556> 2019-01-07 21:14:20.116 7f6343646700 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=CLOSED pgs=0 cs=0 l=0).send_connect_message_reply
-553> 2019-01-07 21:14:20.116 7f6343646700 10 -- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=STATE_CLOSED l=0)._try_send sent bytes 58 remaining bytes 0
-552> 2019-01-07 21:14:20.116 7f6343646700 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=CLOSED pgs=0 cs=0 l=0).handle_connect_message_reply_write r=0
-551> 2019-01-07 21:14:20.116 7f6343646700 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=CLOSED pgs=0 cs=0 l=0).wait_connect_message
-548> 2019-01-07 21:14:20.116 7f6343646700 20 -- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=STATE_CLOSED l=0).read start len=33
-547> 2019-01-07 21:14:20.116 7f6343646700 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=CLOSED pgs=0 cs=0 l=0).handle_connect_message_1 r=0
-544> 2019-01-07 21:14:20.116 7f6343646700 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).wait_connect_message_auth
-543> 2019-01-07 21:14:20.116 7f6343646700 20 -- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=STATE_CLOSED l=0).read start len=158
-540> 2019-01-07 21:14:20.116 7f6343646700 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_auth r=0
-539> 2019-01-07 21:14:20.116 7f6343646700 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2
-538> 2019-01-07 21:14:20.116 7f6343646700 20 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept got peer connect_seq 0 global_seq 17
-537> 2019-01-07 21:14:20.116 7f6343646700 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept of host_type 1, policy.lossy=0 policy.server=0 poli
cy.standby=1 policy.resetcheck=1
-536> 2019-01-07 21:14:20.116 7f6343646700 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept my proto 13, their proto 13
-533> 2019-01-07 21:14:20.116 7f6343646700 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 authorizor_protocol 2 len 158
-523> 2019-01-07 21:14:20.116 7f6343646700 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept setting up session_security.
-498> 2019-01-07 21:14:20.116 7f6343646700 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 existing 0x3719180 on [v2:172.21.15.181:3303/0,v1:172.21.1
5.181:6792/0]
-497> 2019-01-07 21:14:20.116 7f6343646700 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept existing 0x3719180.gseq 0 <= 17, looks ok
-494> 2019-01-07 21:14:20.116 7f6343646700 1 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept connect_seq 0 vs existing csq=0 existing_state=STAT
E_CONNECTION_ESTABLISHED
-492> 2019-01-07 21:14:20.116 7f6343646700 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2 accept connection race, existing 0x3719180.cseq 0 == 0, or
we are server, replacing my attempt
-490> 2019-01-07 21:14:20.116 7f6343646700 10 --2- [v2:172.21.15.74:3302/0,v1:172.21.15.74:6791/0] >> [v2:172.21.15.181:3303/0,v1:172.21.15.181:6792/0] conn(0x3719a80 msgr2 :45244 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).replace accept replacing 0x3719180
-290> 2019-01-07 21:14:20.120 7f6343646700 -1 /build/ceph-14.0.1-2341-ge5f8cd8/src/msg/async/ProtocolV2.cc: In function 'Ct<ProtocolV2>* ProtocolV2::replace(AsyncConnectionRef, ceph_msg_connect_reply&, ceph::bufferlist&)' thread 7f6343646700 time 2019-01-07 21:14:20.123772
/build/ceph-14.0.1-2341-ge5f8cd8/src/msg/async/ProtocolV2.cc: 2242: FAILED ceph_assert(can_write == WriteStatus::NOWRITE)
ceph version 14.0.1-2341-ge5f8cd8 (e5f8cd86d383cb496625e659e768ddbc7a748819) nautilus (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f6353d33079]
2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f6353d33254]
3: (ProtocolV2::replace(boost::intrusive_ptr<AsyncConnection>, ceph_msg_connect_reply&, ceph::buffer::list&)+0xa60) [0x7f6354079bb0]
4: (ProtocolV2::handle_connect_message_2()+0x241b) [0x7f635407c0ab]
5: (ProtocolV2::handle_connect_message_auth(char*, int)+0x138) [0x7f635407ccc8]
6: (ProtocolV2::read_event()+0x10c) [0x7f63540653ac]
7: (AsyncConnection::process()+0x464) [0x7f6354038924]
8: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0x1fdf) [0x7f6354083bff]
9: (()+0x61cbaa) [0x7f635408ebaa]
10: (()+0x8f003f) [0x7f635436203f]
11: (()+0x76ba) [0x7f63529056ba]
12: (clone()+0x6d) [0x7f635212e41d]
/a/sage-2019-01-07_20:29:40-rados:multimon-wip-sage3-testing-2019-01-07-1253-distro-basic-smithi/3434750