Bug #49534
Updated by Sage Weil about 3 years ago
osd.3 is trying to connection to osd.0 and getting disconnected, then retrying, in a loop here is a sample attempt and rejection: osd.3: <pre> 2021-02-27T16:10:00.376+0000 7f4d25fd1700 10 -- 172.21.15.131:0/7 create_connect v2:172.21.15.131:6801/7, creating connection and registering 2021-02-27T16:10:00.376+0000 7f4d25fd1700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_NONE l=1)._connect 2021-02-27T16:10:00.376+0000 7f4d25fd1700 1 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-27T16:10:00.376+0000 7f4d25fd1700 20 Event(0x55f52eabd040 nevent=5000 time_id=6055689).wakeup 2021-02-27T16:10:00.376+0000 7f4d25fd1700 10 -- 172.21.15.131:0/7 create_connect 0x55f5466c1c00 v2:172.21.15.131:6801/7 v2:172.21.15.131:6801/7 2021-02-27T16:10:00.376+0000 7f4d25fd1700 10 -- 172.21.15.131:0/7 connect_to v2:172.21.15.131:6801/7 new 0x55f5466c1c00 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTING l=1).process 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 Event(0x55f52eabd040 nevent=5000 time_id=6055690).create_file_event create event started fd=73 mask=1 original mask is 0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 EpollDriver.add_event add event fd=73 cur_mask=0 add_mask=1 to 5 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 Event(0x55f52eabd040 nevent=5000 time_id=6055690).create_file_event create event end fd=73 mask=1 current mask is 1 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 EpollDriver.del_event del event fd=73 cur_mask=1 delmask=2 to 5 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTING_RE l=1).process connect successfully, ready to send banner 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).read_event 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).start_client_banner_exchange 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._banner_exchange 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 26 remaining bytes 0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=10 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=10 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner r=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=16 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload r=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 1 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 72 remaining bytes 0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main r=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame_segment seg_idx=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=40 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=1 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello payload.length()=36 2021-02-27T16:10:00.376+0000 7f4d420cb700 5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello received hello: peer_type=4 peer_addr_for_me=v2:172.21.15.131:57564/0 2021-02-27T16:10:00.376+0000 7f4d420cb700 5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello getsockname says I am 172.21.15.131:57564 when talking to v2:172.21.15.131:6801/7 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).post_client_banner_exchange 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request peer_type 4 auth_client 0x7ffee9de5900 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 230 remaining bytes 0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main r=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame_segment seg_idx=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=40 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=4 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more payload.length()=36 2021-02-27T16:10:00.376+0000 7f4d420cb700 5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more auth reply more len=32 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 214 remaining bytes 0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main r=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame_segment seg_idx=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=56 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=6 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done payload.length()=52 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 68 remaining bytes 0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main r=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame_segment seg_idx=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=36 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=7 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_signature payload.length()=32 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_signature pre-auth signature success sig_frame.signature()=68cd43d6ef51988f2765412e0dca3701bef36a9917153a20923ff3d262d5d231 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_client_ident 2021-02-27T16:10:00.376+0000 7f4d420cb700 5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_client_ident sending identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=3 global_seq=3252313 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 159 remaining bytes 0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main r=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame_segment seg_idx=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=92 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_segment r=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_dispatch tag=9 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_server_ident payload.length()=88 2021-02-27T16:10:00.376+0000 7f4d420cb700 5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_server_ident received server identification: addrs=v2:172.21.15.131:6801/7 gid=0 global_seq=6505107 features_supported=3f01cfbb7ffdffff features_required=e01020042240000 flags=1 cookie=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=SESSION_CONNECTING pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).handle_server_ident connect success 0, lossy = 1, features 4540138303579357183 2021-02-27T16:10:00.376+0000 7f4d420cb700 1 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame 2021-02-27T16:10:00.376+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 2021-02-27T16:10:00.377+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2021-02-27T16:10:00.377+0000 7f4d420cb700 20 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=32 2021-02-27T16:10:00.377+0000 7f4d420cb700 1 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 73 2021-02-27T16:10:00.377+0000 7f4d420cb700 1 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-02-27T16:10:00.377+0000 7f4d420cb700 20 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main r=-1 2021-02-27T16:10:00.377+0000 7f4d420cb700 1 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-02-27T16:10:00.377+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0)._fault 2021-02-27T16:10:00.377+0000 7f4d420cb700 2 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0)._fault on lossy channel, failing 2021-02-27T16:10:00.377+0000 7f4d420cb700 1 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-27T16:10:00.377+0000 7f4d420cb700 5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).reset_recv_state 2021-02-27T16:10:00.377+0000 7f4d420cb700 5 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 crc :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).reset_security 2021-02-27T16:10:00.377+0000 7f4d420cb700 10 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=READY pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).discard_out_queue started 2021-02-27T16:10:00.377+0000 7f4d420cb700 20 EpollDriver.del_event del event fd=73 cur_mask=1 delmask=3 to 5 2021-02-27T16:10:00.377+0000 7f4d25fd1700 1 -- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 msgr2=0x55f544d0d400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-02-27T16:10:00.377+0000 7f4d25fd1700 1 --2- 172.21.15.131:0/7 >> v2:172.21.15.131:6801/7 conn(0x55f5466c1c00 0x55f544d0d400 unknown :-1 s=CLOSED pgs=6505107 cs=0 l=1 rev1=1 rx=0 tx=0).stop </pre> on osd.0, <pre> 2021-02-27T16:10:00.376+0000 7fb0f6a06700 5 --2- v2:172.21.15.131:6801/7 >> conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_client_ident received client identification: addrs=172.21.15.131:0/7 target=v2:172.21.15.131:6801/7 gid=1 global_seq=3252313 features_supported=3f01cfbb7ffdffff features_required=800000000001000 flags=1 cookie=0 2021-02-27T16:10:00.376+0000 7fb0f6a06700 10 -- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 msgr2=0x559f56e11200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._infer_target_addr 172.21.15.131:0/7 -> 172.21.15.131:0/7 2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).handle_existing_connection existing=0x559f6b37dc00 2021-02-27T16:10:00.376+0000 7fb0f6a06700 1 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).handle_existing_connection existing=0x559f6b37dc00 is a lossy channel. Stopping existing in favor of this connection 2021-02-27T16:10:00.376+0000 7fb0f6a06700 1 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b37dc00 0x559f618c2500 crc :-1 s=READY pgs=3252312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-27T16:10:00.376+0000 7fb0f6a06700 5 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b37dc00 0x559f618c2500 crc :-1 s=READY pgs=3252312 cs=0 l=1 rev1=1 rx=0 tx=0).reset_recv_state 2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 Event(0x559f539754c0 nevent=5000 time_id=4082214).wakeup 2021-02-27T16:10:00.376+0000 7fb0f6a06700 10 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b37dc00 0x559f618c2500 crc :-1 s=READY pgs=3252312 cs=0 l=1 rev1=1 rx=0 tx=0).discard_out_queue started 2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 Event(0x559f539754c0 nevent=5000 time_id=4082214).wakeup 2021-02-27T16:10:00.376+0000 7fb0f5a04700 5 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b37dc00 0x559f618c2500 crc :-1 s=CLOSED pgs=3252312 cs=0 l=1 rev1=1 rx=0 tx=0).reset_recv_state (warped) reseting crypto handlers 2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).send_server_ident 2021-02-27T16:10:00.376+0000 7fb0f6a06700 10 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).discard_requeued_up_to 0 2021-02-27T16:10:00.376+0000 7fb0e5121700 2 osd.0 1419 ms_handle_reset con 0x559f6b37dc00 session 0x559f6a106d20 2021-02-27T16:10:00.376+0000 7fb0f6a06700 5 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).send_server_ident sending identification: addrs=v2:172.21.15.131:6801/7 gid=0 global_seq=6505106 features_supported=3f01cfbb7ffdffff features_required=e01020042240000 flags=1 cookie=0 2021-02-27T16:10:00.376+0000 7fb0f6a06700 10 -- v2:172.21.15.131:6801/7 accept_conn 0x559f6b930c00 172.21.15.131:0/7 2021-02-27T16:10:00.376+0000 7fb0f6a06700 10 -- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 msgr2=0x559f56e11200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 124 remaining bytes 0 2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).server_ready 2021-02-27T16:10:00.376+0000 7fb0f5a04700 5 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b37dc00 0x559f618c2500 crc :-1 s=CLOSED pgs=3252312 cs=0 l=1 rev1=1 rx=0 tx=0).reset_security 2021-02-27T16:10:00.376+0000 7fb0f6a06700 1 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=READY pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=READY pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).read_frame 2021-02-27T16:10:00.376+0000 7fb0f6a06700 20 -- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 msgr2=0x559f56e11200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=32 ... 2021-02-27T16:10:00.376+0000 7fb0f5a04700 10 -- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6be2e000 msgr2=0x559f6c184a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._infer_target_addr 172.21.15.131:0/7 -> 172.21.15.131:0/7 2021-02-27T16:10:00.376+0000 7fb0f5a04700 20 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6be2e000 0x559f6c184a00 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).handle_existing_connection existing=0x559f6b930c00 2021-02-27T16:10:00.376+0000 7fb0f5a04700 1 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6be2e000 0x559f6c184a00 crc :-1 s=SESSION_ACCEPTING pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).handle_existing_connection existing=0x559f6b930c00 is a lossy channel. Stopping existing in favor of this connection 2021-02-27T16:10:00.376+0000 7fb0f5a04700 1 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=READY pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-27T16:10:00.376+0000 7fb0f5a04700 5 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=READY pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).reset_recv_state 2021-02-27T16:10:00.376+0000 7fb0f5a04700 20 Event(0x559f53975040 nevent=5000 time_id=2045227).wakeup 2021-02-27T16:10:00.376+0000 7fb0f5a04700 10 --2- v2:172.21.15.131:6801/7 >> 172.21.15.131:0/7 conn(0x559f6b930c00 0x559f56e11200 crc :-1 s=READY pgs=3252313 cs=0 l=1 rev1=1 rx=0 tx=0).discard_out_queue started </pre> i.e., we are disconnected because another connection comes in from the same peer. that second connection has the same seq! this is reproduced by rados/cephadm/thrash with podman --init arg enabled.